Automattic JavaScript Engineer application by Paul D. Fernhout

Contents

An email to another Automattician on why Slack is problematical

This is an email I sent to an Automattician who I had been having a side discussion with related to my application and other topics. They raised some good points in defense of Auotmattic’s use of Slack, and also raised concerns (perhaps valid) that I was being too much of a purist or perfectionist on the Slack issue. That person may well be right. :-) And I may well end up using Slack somewhere else soon enough, as they also pointed out that realistically it may even now and certainly soon be hard to get a job anywhere without using Slack.  I appreciate the thought and concern they put into trying to help me. Below was my reply.

Sent: 2015-22-27 (with a few typos fixed)

Some responses to try to address the points you raise.

–Paul Fernhout

Outward facing software for developers is different

When Automattic uses Zendesk, Salesforce, or Trello, Automattic does not force free software developers working on GPLd WordPress software to sign up for a proprietary service and agree to a third-party Terms of Service that can included terms like the following (for the API):
https://slack.com/terms-of-service/api?nojsmode=1

“Functionality. You may not use the Slack API to replicate or compete with core products or services offered by Slack. You acknowledge and agree that Slack has or may in the future offer products or services that are similar to your Application, and nothing will prevent Slack from doing so;”

That can be read as, whatever Slack develops, Automattic has to stop supporting the analogous capability in WordPress through the Slack API. So, when (probably not “if”) Slack starts supporting instant blogging on public-facing websites, Automattic may be on shaky legal ground to integrate WordPress with Slack as it could be seen as “competing with” a Slack core product or service.

That current TOS could perhaps even be read (in an extreme case) as any software patents (evil things, true) that you might hold would not be able to prevent Slack from making the same software if you ever use their API. Meanwhile, Slack could use any software patents it holds to shut down WordPress.com.

[Added later: An even worse reading of this API TOS is that you can’t use Slack even to discuss building alternatives to Slack on WordPress sites that integrate with Slack because you are indirectly using the Slack API to “compete with … Slack”. True, that is a very broad and probably unintended and maybe legally indefensible reading of the API TOS. But see how quickly what you can now talk about inside Automattic depends on careful reading and analysis of third-party legal documents prepared by a potential competitor? So, discussing hiring someone (perhaps me) to make real-time tools for Automattic with the discussions posted internally on Automattic P2/O2 blogs that link to Slack may indeed, in that most broad reading, be a violation by Automattic of Slack’s TOS, meaning Slack is perhaps within its legal rights to immediately disconnect all Automatticians from Slack and prevent any further access, even to retrieve existing content. Of course, Slack can do that anyway, even without Automattic violating the TOS, as they say: “We may change, suspend or discontinue the Slack API and suspend or terminate your use of the Slack API, the Slack Service, and/or Slack Brand at any time and for any reason, without notice.” That is a problematical legal guarantee to put at the core of a business like Automattic and at the heart of a free software community like WordPress.org.]

Remember, at *any* time Slack can change these TOS to be *even* worse for Automattic or any other developers. At *any* time. And there is nothing much Automattic can do about that other than suddenly stop using the service.

[Added later: Slack specifically says: “Modification of the API TOS. We may change, add to or delete this API TOS or any portion thereof from time to time in our sole discretion. If we make a material change to this API TOs, we will provide you with reasonable notice prior to the changes either by emailing the email address associated with your account or by posting a notice on the Developer Site. You acknowledge that these updates and modifications may adversely impact how you access, use, and communicate with the Slack API. If any change in unacceptable to you, then your only recourse is to cease all use of the Slack API. Your continued access or use of the Slack API will mean that you agree to the updates and modifications.”]

[Slack says essentially the same thing about use of the Slack service itself in the Slack (non-API) TOS, where “your only remedy is to cancel your account and stop using the Services”. Even if you disputed that, you are also legally bound to arbitration by the TOS, not a court case, and the TOS also makes class action lawsuits impossible as well.]

[Meanwhile Slack has a record of all your communications, including any of legal plans regarding Slack. As Slack says in their privacy policy, if they just assign random identifiers to users in communications they look at, they can (depending on how you read the TOS) then review all of Automattic’s internal communications and use it however they want (even assuming they have to go through the motions of “privacy” even that far in practice). They say: “To make the product better we have to understand how users are using it. We have a fair bit of data about usage and we intend to use it many different ways to improve our products, including research. This policy is not intended to place any limits on what we do with usage data that is aggregated or de-identified so it is no longer tied to a Slack user.”]

[As Slack says in a recent job posting for a Data Analyst: “Slack is looking for experienced analysts to help drive the understanding, growth, and success of our business. The right candidates will be comfortable working with a nascent data infrastructure to identify, monitor, and impact key metrics and help us make better decisions. They will be scrappy self-starters who are excited about having a big impact from a new team.” So, Slack is now hiring fishers of interesting data fish in the newly-created Slack ocean. Previously Slack did a deep dive into what emojis were used by Slack users. It might benefit Slack’s “research” into improving Slack to do a deep dive into all of Automattic’s internal discussions of Automattic’s and WordPress’ strengths, weaknesses, and plans as well. That all may be perfectly acceptable legally depending on how you read the Slack TOS, assuming “Slack user” refers to individual users and not businesses. Would it in any way be in disagreement with the Slack TOS if such a Data Analyst could just aggregate the top themes and key phrases of Automattic’s internal communications, which is probably all a Slack CEO has time to look at anyway? Even if that was against the Slack TOS today (arguable, but unlikely IMHO), how about tomorrow when Automattic is even more dependent on Slack and has added even more “ephemeral” content to Slack instead of on Automattic’s own P2/O2 internal blogs? As Automattic says, “Communication is Oxygen”. Now that Automattic has chosen to swim in the Slack data ocean, Slack, a proprietary competitor in the web communications space, is now in charge of a substantial (and ever growing) volume of Automattic’s oxygen supply.]

The moral analogy

I added the next paragraph to the Automattic application page I had written, along with a list of Slack alternatives (in a new “probiotics” section), in part from reflecting on points you raise in your email:

As an analogy, consider Seed Saver’s Exchange whose mission is “Saving America’s Heirloom Seeds”. Such seeds are free and open source in the sense that they are not covered by copyrights or patents, and you can save your seeds from this year’s crop and grow from them the next year’s crop (which you can’t legally do with a lot of commercial seeds now). As they say in their story, “Diane and Kent went on to form a network of gardeners interested in preserving heirloom varieties and sharing seeds. Today, with 13,000 members and 20,000 plant varieties, Seed Savers Exchange makes its home on 890 scenic acres in Winneshiek County, Iowa, at Heritage Farm.” Now, imagine if Seed Savers decided to “improve” those acres to make them more “open” to stakeholders who were coming for a big get together to create the future of Seed Savers. Imagine, to spruce up the place, Seed Savers replaced all the flowers by the main buildings with commercial proprietary flower varieties because they were easier for gardeners to get and cheaper to buy. Imagine they served only GMO produce from Monsanto at the get-together because it was likewise cheaper and easier to get and such food had some advantages claimed by Monsanto (a major developer of proprietary GMO seeds). Imagine they even announced a big partnership with Monsanto at the get together and on the CEO of the organization’s blog. And, imagine Seed Savers paraded these decisions in front of stakeholders of evidence of how hip and cool and “open” and heirloom-gardener-friendly they were being, and said that using Monsanto products at the heart of their operation did not matter to their mission because they were still growing heirloom seeds in the actual fields. And imagine people came along to suggest trying new varieties of free and open heirloom seeds to meet whatever needs Monsanto was supposedly meeting and just got told there was not enough time or money to consider that. How many stakeholders do you think would walk away from that stakeholder meeting shaking their heads in bewilderment? Or worse?

There are some links in there, like this one as to why people think Monsanto is doing bad stuff:
https://www.organicconsumers.org/news/seeds-evil-monsanto-and-genetic-engineering

Still, Automattic may well change its Slack usage to something FOSS down the road. It will be painful socially and expensive, but it may happen in a couple of years if this plays out problematically for Automattic. Still, the Slack developer community may eventually push back and Slack may revise its TOS to be more friendly to developers, like at least allowing competition with the services they offer. We’ll see.

I hope that what I’m saying is all nonsense and never affects Automattic negatively. But the risk remains. Of course, whether to take any risk is a business decision as life is full of both risk and opportunity. But, when you build your business around developing and supporting free and open source decentralizable communications software (as Automattic has with the GPL-d WordPress), a negative community reaction to forcing FOSS developers to start using proprietary centralized software with restrictive TOS for reasons of short-term convenience may be another risk to consider as well. Thus the Seed Savers analogy above.

I’m not talking about perfection. Automattic’s choices of React or JavaScript vs. Mithril or TypeScript I can accept even if I might make them differently. Stuff like whether Automattic insists people take at least four weeks vacation at a minimum (as another random suggestion of many in my application) likewise is just a thought for discussion. But this Slack issue goes to the core of what Automattic does as the main part of its mission — FOSS communications software to make the web a better place. And the point is that Automattic has made a proprietary choice that has moved the WordPress developer community backwards as far as freedom — rather than figuring out a free-er way to address whatever unmet needs Slack addresses.

This Slack issue is fundamentally about sustaining mutual respect related to Automattic engaging directly with the moral convictions that citizens of the WordPress community (especially developers) bring to their involvement with community and the software (to borrow some wording from the end of the Michael Sandel video on democratic debate linked below).

Automattic has plenty of resources to explore a huge real-time space given the upside potential

Automattic just got US$160 million in investment. Most of what Slack is spending, it is spending on marketing [and sales]. With several Slack clones out there already (as you point out) the cost to integrating Slack-like functionality and more into WordPress can’t really be that high (a few million at most?) to make something at least minimally acceptable — or perhaps, even better :-) Automattic does not have to cover all use cases Slack does even — just the ones most essential to a healthy democracy or to businesses that choose to be open and engaging to their customers, suppliers, and funders. Software like Mattermost btw already has mobile integration, so that is not a real showstopper. Of course, if Automattic went down this path, it would find itself on the wrong side of US$340 million invested in Slack (and probably more to come soon when Facebook or whoever buys it) not to mention US$80 million to be given out to Slack developers.

Still, the alternative to creating real-time software for WordPress seems to be for Automattic to completely abandon the real-time communications market (and perhaps more depending on future changes to the Slack TOS). That is problematical for Automattic’s future survival given that real-time interaction is a *huge* upcoming market, especially as it ties closely into mobile support. It is not just chat, or even voice, but whiteboards, collaborative drawing, websurfing together, structured arguments, online games, virtual reality, and a whole bunch of other stuff. So, I feel the choice for Automattic to cede the real-time communications market to Slack is a big strategic mistake, and not just because real-time support is the kind of thing I wanted Automattic to hire me to work on — answering your implied question of who would do that kind of work. :-)

BTW, see from the other day for a big new market about to emerge in real-time communications:
“Virtual Reality’s Killer App: Taking Over Your Social Life”
https://soylentnews.org/article.pl?sid=15/12/25/0733205

“The door to mass-market virtual reality is about to burst open. Engineers have solved most of the hardware challenges, driven down the price to just a few hundred dollars, done extensive testing, and gotten software tools into the hands of creative developers. Store shelves will soon be teeming with head-mounted displays and hand controllers that can paint dazzling virtual worlds. And then the first wave of VR immigrants will colonize them. You might think the first adopters will be gamers, but you’d be wrong. The killer app for virtual reality will more likely be something to enhance ordinary social experiences—conversations with your loved ones, a business meeting, a college class—but carried out with a far richer connection than you could establish by texting or talking or Skyping.”

Automattic has essentially conceded real-time stuff like that as well to Slack. I might not care that much if Slack was FOSS and Slack articulated values similar to Automattic, but Slack is proprietary and the company does not articulate values like making the web a better place.

So, I feel WordPress needs to get minimal real-time support to keep progressing if it wants to take on the other 75% of the web and more with FOSS. And it is not that hard — just use socket.io that was developed by Guillermo Rauch, an Automattician! :-) Put him in charge of the whole thing! :-) Well, Automattic would need socket.io along long with a ping server to tell clients when to poll a WordPress webserver in some cases, as I suggested would make a good trial project for me. :-) But Guillermo might be the best person to lead such an effort none-the-less. I’m sure I could have learned a lot from him and had a lot of fun along the way, too.

Cost of switching is why proprietary services are problematical

As far as Automattic’s new cost/debt issues moving away from a proprietary service like Slack, that is the exact reason to avoid such proprietary services in the first place. :-) Everything you are suggestion along those lines of a big cost of switching is, to me, proof of what I’m saying is true. :-) And that’s the reason a lot of people use FOSS WordPress over Wix or whatever as a primary community communications platform, even, and perhaps especially, if they host at WordPress.com. :-)

Cognitive dissonance makes it hard to see mistakes

Still, as is said in this book on cognitive dissonance and decision making:
“Mistakes Were Made (But Not by Me): Why We Justify Foolish Beliefs, Bad Decisions, and Hurtful Acts”
http://www.amazon.com/Mistakes-Were-Made-But-Not/dp/0151010986

“Why do people dodge responsibility when things fall apart? Why the parade of public figures unable to own up when they screw up? Why the endless marital quarrels over who is right? Why can we see hypocrisy in others but not in ourselves? Are we all liars? Or do we really believe the stories we tell? Renowned social psychologists Carol Tavris and Elliot Aronson take a compelling look into how the brain is wired for self-justification. When we make mistakes, we must calm the cognitive dissonance that jars our feelings of self-worth. And so we create fictions that absolve us of responsibility, restoring our belief that we are smart, moral, and right—a belief that often keeps us on a course that is dumb, immoral, and wrong. Backed by years of research and delivered in lively, energetic prose, Mistakes Were Made (But Not by Me) offers a fascinating explanation of self-deception—how it works, the harm it can cause, and how we can overcome it.”

So, now that I walked away from an (20% chance based on hiring stats) of a mostly-work-from-home six-figure job at Automattic developing open source (where my wife is going to be tremendously upset with me when she finds out why [did not want to spoil Solstice/Christmas/NewYears for her], rightfully so considering our current debt load from spending a year writing the FOSS NarraFirma as well as our choice to continue to homeschool our kid meaning being around home is a big deal, especially in a rural area with long commutes), given all that, I’m naturally biased to defend that choice as above. :-) I’m tremendously biased at this point. So, I can’t say I’m going to be easily persuadable by your arguments, even the most practical and sensible ones. :-)

Arguing together productively in a health way can help overcome cognitive dissonance

I can still acknowledge this though, by Hugo Mercier:
“Researcher Responds to Arguments Over His Theory of Arguing”
http://artsbeat.blogs.nytimes.com/2011/06/15/researcher-responds-to-arguments-over-his-theory-of-arguing/?_r=0

“The main idea of the “argumentative theory of reasoning,” put forward by Dan Sperber and myself is that the function of human reasoning — why it evolved — is to improve communication by allowing people to debate with each other: to produce and evaluate arguments during a discussion. This contrasts with the standard view of reasoning — apparently shared by quite a few of the readers — that reasoning evolved in order to further individual reasoning: to make better decisions, to plan ahead, to get better beliefs, etc. We have gathered a lot of evidence in support of our theory. The interested reader may enjoy a short summary, and the bravest may read the main academic article (use the “One-Click Download” link on the summary Web page). For those who don’t have the time or the inclination, let me simply try to correct an important but common misconception.

We do not claim that reasoning has nothing to do with the truth. We claim that reasoning did not evolve to allow the lone reasoner to find the truth. We think it evolved to argue. But arguing is not only about trying to convince other people; it’s also about listening to their arguments. So reasoning is two-sided. On the one hand, it is used to produce arguments. Here its goal is to convince people. Accordingly, it displays a strong confirmation bias — what people see as the “rhetoric” side of reasoning. On the other hand, reasoning is also used to evaluate arguments. Here its goal is to tease out good arguments from bad ones so as to accept warranted conclusions and, if things go well, get better beliefs and make better decisions in the end.

Many people seem to attribute to us the view that reasoning is mostly used to trick people with deceptive arguments into accepting bogus conclusions (“manipulating others”). We have not made this claim, and it is in fact not evolutionarily plausible. Such an outcome would run against the logic of the evolution of communication, which can be explained very simply. For communication to be possible in a species, it has to benefit both speakers and listeners. If the speakers didn’t benefit, they would have evolved to stop speaking; if the listeners didn’t benefit, they would have evolved to stop listening. The same is true of argumentation. If the people who listen to arguments were not better off on average, they would evolve to stop listening to arguments.

Once this misconception is dispelled, it should become clear that we are not defending any kind of relativist position. There are conclusions that are more or less true and argumentation, when it works well, allows people to share the best conclusions with others. …

Finally, and maybe most importantly, we are not saying that people should not use reasoning. People should use reasoning. But we claim — and provide a substantial amount of evidence in support of this claim — that when people reason, they are usually better off reasoning with people who disagree with them rather than with like-minded peers or on their own.”

Let’s hope that by arguing together like this we are providing the groundwork for better decisions in the future by the community. :-) And maybe someday better tools for the WordPress community to argue together in a healthy and productive way. :-)

Also related:
“The lost art of democratic debate”
https://www.ted.com/talks/michael_sandel_the_lost_art_of_democratic_debate

“Democracy thrives on civil debate, Michael Sandel says — but we’re shamefully out of practice. He leads a fun refresher, with TEDsters sparring over a recent Supreme Court case (PGA Tour Inc. v. Martin) whose outcome reveals the critical ingredient in justice.”

As Michael Sandel says at the very start, “One thing the world needs, on thing this country desperately needs is a better way of conducting our political debates. We need to rediscover the lost art of democratic argument.”

Some related humor: :-)
“Argument Clinic – Monty Python’s The Flying Circus”
https://www.youtube.com/watch?v=kQFKtI6gn9Y

Communities need some level of diversity and related healthy conflict

This next video mentions a historical view on changes in dealing with conflict (from a traditional view of avoiding conflict, to a behavioral/contemporary view that conflict is inevitable and so manage it, and then to an interactionist view of conflict that conflict is necessary for growth and innovation and should even be stimulated to avoid stagnation if there is not enough of it especially in the context of a diversity of ideas):
“Conflict Management in the Project Environment: Views and Causes”
https://www.youtube.com/watch?v=CAUooBKKFl0

A community that tries too hard to be “nice”, to avoid the healthy version of conflict, may always risk ending up in this situation:
“Smile or Die”
https://www.youtube.com/watch?v=u5um8QWWRvo

“Acclaimed journalist, author and political activist Barbara Ehrenreich explores the darker side of positive thinking.”

“Difference” is powerful as diversity, but that means conflicts need to be dealt with in a healthy and productive way:
“The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies”
http://www.amazon.com/Difference-Diversity-Creates-Schools-Societies/dp/0691128383

“In this landmark book, Scott Page redefines the way we understand ourselves in relation to one another. The Difference is about how we think in groups–and how our collective wisdom exceeds the sum of its parts. Why can teams of people find better solutions than brilliant individuals working alone? And why are the best group decisions and predictions those that draw upon the very qualities that make each of us unique? The answers lie in diversity–not what we look like outside, but what we look like within, our distinct tools and abilities.

The Difference reveals that progress and innovation may depend less on lone thinkers with enormous IQs than on diverse people working together and capitalizing on their individuality. Page shows how groups that display a range of perspectives outperform groups of like-minded experts. Diversity yields superior outcomes, and Page proves it using his own cutting-edge research. Moving beyond the politics that cloud standard debates about diversity, he explains why difference beats out homogeneity, whether you’re talking about citizens in a democracy or scientists in the laboratory. He examines practical ways to apply diversity’s logic to a host of problems, and along the way offers fascinating and surprising examples, from the redesign of the Chicago “El” to the truth about where we store our ketchup.

Page changes the way we understand diversity–how to harness its untapped potential, how to understand and avoid its traps, and how we can leverage our differences for the benefit of all.”

Better tools to support handling workplace conflict in real-time connect to all that, but I’d much rather see such tools be FOSS and developed by Automattic (or another FOSS vendor) than be proprietary and developed by Slack (or dependent on Slack if made by third-parties) or some other proprietary-emphasizing company.

Still plugin’ along on Twirlip/Pointrel :-)

BTW, I’m still working towards butter FOSS tools to support arguing together productively in real-time. :-) For example, I just added displaying a restricted subset of inline HTML (including images) to Twirlip: :-)
https://github.com/pdfernhout/Twirlip2/blob/chat_messages_with_images/webapp/source/plugins-for-local-libre-or-standard-services/chat-text/chat-text-plugin.ts

Don’t know how long I can keep that up, but I’m trying anyway…

New Year’s wishes and thank you

Hope you had good Christmas or other holidays, and that you (and Automattic) and me too have a great New Year with many healthy and productive and interesting arguments and conflicts. :-)
https://www.google.com/search?q=dealing+with+conflict+youtube

And I hope that all our arguments and conflicts with anyone are never depressing, even though that can be useful sometimes. :-) For example:
https://en.wikipedia.org/wiki/Evolutionary_approaches_to_depression#Analytical_rumination_hypothesis

“This [analytical rumination] hypothesis suggests that depression is an adaptation that causes the affected individual to concentrate his or her attention and focus on a complex problem in order to analyze and solve it.”

All the best. Beyond leading to some improvements in that long document, our discussions have made this situation a lot more pleasant for me, even when we disagree. Thank you. :-)

–Paul Fernhout
http://www.pdfernhout.net/
====
The biggest challenge of the 21st century is the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.


A sad letter to write regarding my feelings about Slack and Automattic’s mission

tl;dr Automattic’s choice of Slack for internal and external communications and promotion of it on WordPress.org is in conflict with its stated mission to make the web a better place (implicitly via FOSS and related support services). I’m willing to throw away this potentially valuable life-improving job opportunity at an otherwise awesome company rather than become a supporter of that questionable decision because I believe in the value of that mission. So, I feel the best way I can help Automattic right now to reflect on its mission and hopefully reaffirm it eventually is to say “no” to Slack, as otherwise that would be enabling the abandonment of that important mission. If that situation changes such as Automattic moves to the FOSS Mattermost Slack-clone or something else, I’d be happy to reapply. There are other possible ways we might still work together as well. I know this one instance of refusal won’t make much of a difference, but hopefully if a lot of other people do the same, Automattic might move back to FOSS and/or standard solutions for communications.

Nikolay-

I really like Automattic and its past accomplishments and current technical directions like with Calypso. Automattic has done some awesome things for the world. However, I guess Slack is indeed a blocker, sorry. I had originally expected, like from reading Scott Berkun’s book, that Automattic would be fully investing in free and open communications technologies given a stated commitment to making the web a better place (even as I’ve noticed more and more Slack references related to Automattic over the past couple of months). I am sad and disappointed to see the growing use of Slack in Automattic and with WordPress.org because I feel that undermines the FOSS community that has helped WordPress and Automattic succeed by forcing them to use proprietary communications software to get their basic communications needs met. I feel the WordPress community (and I as part of it) need Automattic to reaffirm its commitment to making the web a better place by emphasizing creating and using non-proprietary communication tools like WordPress and IRC and email and similar. I know it will be very hard for Automattic to do that at this point. I am sorry I did not apply a couple of years ago to hopefully help encourage Automattic to stay truer to its mission.

How can that conflict be resolved in a health way at this point? Some options are working together in some other ways to make Slack-replacement WordPress plugins, Automattic going back to IRC maybe with an easier to use web front end, Automattic going forward to something like Y-Combinator-funded FOSS Mattermost, just parting ways for now and maybe reconnecting when things change, or maybe something I have not thought of?

While I might like Automattic overnight to reverse the Slack decision and purge Slack from its processes, I know that probably is not realistic. And I’m also realizing that even if I could persuade you personally to consider something else just for me regarding a chat just for this interview, with the expectation of subsequent interviews, a trial, and a Matt chat, I’d have to start using Slack before I was hired by Automattic anyway, again to be realistic. It would also use up a lot of scarce Automattic bandwidth just to discuss Slack further with each and every person in Automattic I had to interact with. So, I have to admit your preference to use Slack is reasonable for Automattic hiring given the current situation, even if it conflicts with my idealized expectations about Automattic or my desire to avoid Slack specifically.

That conflict does give me reason for pause and reflection, mostly in regards to what the choice of Slack says about Automattic’s current cultural direction and how I might fit into it. Reflecting on that, it suggests to me that maybe it is not going to be a good match between me and Automattic’s current directions regardless of what you yourself were willing to consider for chatting? I could easily get a Slack account in two minutes and continue the Automattic job application process anyway (I need to find work that pays again, and even using Slack, Automattic is a better company than probably almost any other). While on principle I don’t want to get a Slack account, there are always multiple principles to choose from, :-) and other principles like loyalty to family and obligations to creditors recommend making such a compromise.

But given the above, it would probably not be right for me in the long-term to do so that, given my strong feelings about this, including how  much I feel Slack is harmful to both Automattic and the WordPress community. I don’t feel it would be right for Automattic either to have an employee who felt pressured into such a decision. So, in the long-term, I don’t think my getting a Slack account to get a job at Automattic to create WordPress-based real-time collaboration tools (or whatever I might do of use with Automattic) would be a good idea, even if it is in some sense a quick fix to a surface aspect of this conflict as well as my need to keep our bills paid.

The deep issue is that Automattic has chosen a proprietary platform for its communications and encourages WordPress.org users to use it, and that choice seems to me to be incompatible with Automattic’s stated ethos, and it is the ethos I was looking for more than the specifics (as good as they are). I really don’t understand how Automattic could make such a choice for itself, let alone WordPress.org.

A statement on WordPress.org relating to Slack being “open” is, if not a lie, clearly a half truth: https://make.wordpress.org/chat/

“The WordPress project uses Slack as its main real-time communication platform, replacing all previous communication platforms such as IRC and Skype. When compiling a list of things people liked about our previous communication methods, it was obvious that Slack was all of that and more, including: Open for everyone”

Slack is not “open for everyone” unless you have completely redefined open to mean “put all your data in a proprietary centralized service you eventually need to pay monthly to use”.

As someone who has been developing FOSS on-and-off for about two decades, and who recently spent a year full-time writing a WordPress plugin, it is hard to calmly state my feelings reading that kind of statement about being “open”. It even suggests to me that my personal investment in WordPress was even terribly mistaken (even having seemed reasonable at the time started under different WordPress.org policies).

More details are below on that situation, including a list of practical reasons Automattic is at risk business-wise for using Slack, more details on at least one existing Slack alternative (Mattermost), and also more details on that suggestion about how we might work together anyway to develop WordPress plugins to hedge Automattic’s bet on Slack. I tend to think those suggestions won’t change anything at Automattic any time soon (Automattic is busy as it is, probably why Slack seemed like an easy answer), but I felt I should offer them as possibilities at least. If you end up getting 100s of pushbacks on Slack (and frankly, I doubt you will though) then you might go back and revisit them. At least I know you have them. I would not have bothered if I did not feel Automattic has been a great company, and has done a lot for the world, and still has amazing potential.

All the best. And I’ll keep looking out for if Automattic changes its policies back to open standards and/or FOSS implementations (like Mattermost or improved WordPress plugins). If so, and if appropriate with any other obligations and responsibilities I might have then, I’d be eager to re-apply and might hope then I could jump the queue a bit back to this initial interview part. :-)

Anyway, back to coding/inventing a non-proprietary future even if it is folly. :-) Even if I may likely fail at creating such communication tools on my own, I can hope the FOSS community (as with Mattermost and their own take on things, one I might even join my own efforts up with perhaps at some point) can eventually somehow deal healthily with the oncoming well-funded pandemic of Slack invitations. :-) Including by discussing it in depth on (hopefully non-astroturfed) WordPress blogs. :-)

–Paul Fernhout

== More details

One possibility I would enjoy a lot is that Automattic could fund me and others to make WordPress plugins that are alternatives to Slack for small groups. :-)  However, give so many great ideas and good FOSS WordPress plugins developers out there, and several explicit Slack clones out there already (e.g. Mattermost), I know any such suggestion is a tough sell, but I just thought I’d mention it as a Slack alternative as a way to move forward despite this conflict. Although no doubt Matt or others at Automattic would probably want to discuss such an idea using Slack, so back to square one. :-)

Another more likely possible way to resolve the conflict would be for Automattic to pick an existing FOSS system for integrated IM communications (like Mattermost or whatever) as an alternative to Slack. From Mattermost’s site: http://www.mattermost.org/
“Mattermost is “Slack-compatible, not Slack-limited”, supporting a superset of Slack’s incoming and outgoing webhook integrations, including compatibility with existing Slack integrations. From your existing Slack teams, you can import users, public channel history and even theme setting colors into Mattermost. … Mattermost is written in Golang and React, and ships stable compiles releases monthly on the 16th under an MIT license, which then ship in GitLab omnibus monthly on the 22nd. Installs in one-line with Docker, and many install guides are available (including native installs to Ubuntu and Debian, as well as Docker, Cloud Foundry, Heroku installers). Read more about Mattermost features. See also community integrations and applications (including IRC, Hubot, Giphy, Jenkins, GitLab, SVN, RSS/Atom and many others).”

As Mattermost says regarding its origins:
http://www.mattermost.org/why-we-made-mattermost-an-open-source-slack-alternative/

“In 2013 (before Slack), we relied on another popular team messaging service, which became plagued with issues after it was sold to a large company. … In 2014, Slack became popular in Silicon Valley, and our company adopted it for messaging. But we had a problem – our archives were in our old messaging app. After our subscription expired, the old app wanted us to pay them to access our own data (and export still didn’t work!). We hated being locked in. We fumed. Our discussions, our research, our analyses, and gigabytes of our carefully tagged in-game artwork was all held for ransom by the service we had trusted. Slack was good, but it was another proprietary SaaS app, and we’d just been burned. So we decided to build our own. … Along the way we decided to open-source our work. We began 2015 with a re-write in Golang and React. This “Version 3” had a simplified architecture and a web UI working across PCs and phones. … We renamed the project “Mattermost” to emphasize the importance of communication. We believe that 1) internal messaging–across PCs and phones, with file sharing, archiving and search–is a vital layer of IT in a “post-email” world, 2) the platform for internal messaging needs to be open, and 3) that we wanted Mattermost to be the leading project to serve as the open layer for internal communications. …”

Does Mattermost really work? Are there even better ones out there? I don’t know. But as they suggest, even when you might think communications are ephemeral, it is easy to get caught up in relying on such systems as you use them for more and more things, with a risk growing over time. Slack has an export function today apparently. Will it have one tomorrow?

This conflict is not unexpected, or, to be clear, not due to anything specific you or anyone at Automattic wrote; in a way, it’s not really about Slack itself, in the sense that getting an account I use only to talk to Automattic and maybe some other businesses in the future is just a minor risk for me (after all I have a Skype account and they do instant messaging etc.). I can probably even delete the Slack account after we are done chatting to avoid being affected by likely future TOS changes.

But on top of some other concerns about fit, this conflict is mainly what Automattic’s move to Slack as a company says to me about the company and its direction as far as whether various FOSS communications/collaboration-related projects I would like to do would be a good fit at Automattic (as well as just fit between us in general). I could kind of ignore that slow-developing issue until this moment based on reading Scott Berkun’s now outdated book and hope for something to change regarding Automattic and Slack and its FOSS ethic. I first got interested in Automattic when it still used IRC and Skype in 2014, but NarraFirma took forever to finish, otherwise I could maybe had advocated against Slack inside Automattic a year ago. But I guess now I can’t ignore that proprietary Slack issue anymore, as much as I still very much like a lot about Automattic and feel it is doing a lot of things right.

Thank you for your time and all your good work making WordPress work well and be secure. We recently upgraded NarraFirma.com to 4.4, thanks! :-) I hope we can connect again down the road if Automattic as a company ends up moving back to using and promoting non-proprietary communication systems externally and internally (like based on WordPress technology or various internet standards). Though, business-wise, I can expect that will not happen anytime soon.

Earlier today, after you sent your email but before I had seen it, I updated the beginning of that application document, including with some security and business risk questions suggestion why Slack is problematical for Automattic. I included those question below just for your reference as current WordPress “Security Czar”.

Further, while it’s not on that list, another risk is that Slack may become a dividing line with technical recruitment as well as WordPress community participation, as with this email. But Automattic has such a broad base to choose from, in practice, that may never be noticed, especially including as the effects are subtle.

All the best. And again, I appreciate your time and that of others at Automattic, and I wish you all a tremendous lot of good fortune and success. Calypso in particular is an outstanding success and a tremendous step in the right direction for both Automattic and the WordPress community. Calypso’s approach opens up a world of possibilities for Automattic and WordPress users.

So, perhaps two steps forward with Calypso, even if perhaps with one step back with Slack? So, good progress for Automattic overall. :-)

And thank you for supporting Guillermo Rauch’s work on socket.io at Automattic, which may well revolutionize web communications as it sees broader and broader adoption. It’s an amazing innovation in developer ease-of-use to create awesome real-time user experiences. Would not surprise me if Slack uses it? :-)

I hope ideas in the application document I wrote prove useful to Automattic down the road whoever implements them and however they are discussed. Thanks for inspiring me to write them down. :-)

Now I just need to figure out how to tell my wife about this after putting so much time into this effort, while facing a pile of credit card balances piled up from previous FOSS work, as well as probably long commutes and so on for alternatives. :-) And possibly, ironically, then eventually ending up working for some company that adopts Slack for everything but does not have Automattic’s many qood qualities. :-)

However, as I still try to think about this even is great sadness over the situation which neither of us can really fix (since it is a deep social issue and Automattic has already made a commitment to Slack), here might still be other ways we all could work together though. I mainly just want to develop more great JavaScript/TypeScript-based WordPress-compatible plugins, especially ones that support activities even better than what Slack supports, using Automattic’s socket.io. I hope to do that anyway, whether with Automattic or not. Maybe Matt ( perhaps via his https://audrey.co/ ) might even be interested in investing in such plugins to grow WordPress in new directions to get another 25% of web users? Now that Slack has proven there is a hunger for web-based instant communications options and others have shown the value of web-technology-based email clients and such, that seems like a good growth area for competition. I see there that Matt invested in Bindle (“Modern-day IRC), but that looks proprietary and is not for WordPress? Such WordPress plugins might not satisfy all Slack users, but even if we could reach, say, 10% of such users and bring them back to FOSS/WordPress, that could be a big win. :-)

I suggested elsewhere that Automattic could invest in tools to translate React to Mithril automatically as backup option in case of software patent issues, as a way to hedge the risk of that decision. Similarly, creating a WordPress plugin that could work like Slack (and maybe even import what Slack exports) could be a backup option for Automattic if using Slack does not work out for some reason, as a way to hedge the risk from the Slack decision. I know in practice you all are probably too busy to consider such new ventures on top of everything else you do, but just something to think about. Obviously, developing such tools in advance of the need is a better hedge than waiting until a crisis and there is not much time to make them work well.

So, an example of one possibility, Audrey Capital (or Automattic) could, as a form of insurance, invest in my creating WordPress plugins using socket.io and a Node.js ping server (along the lines of what I’ve put in my job application) that can do much of what Slack can do and even more and better (including decision support and sensemaking, and other things like in that Twirlip2 project) — as a hedge against needing to reverse the Slack decision at some point. Frankly though, I don’t have a “business model” besides help make WordPress/Automattic succeed and grow interest in WordPress.com hosting alongside WordPress.org local hosting. :-) Still, I could do a lot in that direction as FOSS just on $100K (maybe even stretching that towards two years work full-time if my wife brings in some income), and even though that is a lot of money, it still might be cheap insurance for hedging a bet on a third-party technology Automattic has essentially bet the company on. :-) And like insurance, you could hope you never need to use it. :-) The same is true for data backups, even though they can be costly to make and verify. Wouldn’t it be worth it for Automattic to *both* hedge its Slack bet and also grow WordPress? And then you don’t have to make a decision about hiring someone who is, admittedly, as weird as me, but we can still be working side-by-side on a common mission of making the web a better place. :-)

Still, while that would be a lot of fun for me, realistically, Automattic switching to Mattermost or whatever would probably be cheaper and easier.

–Paul Fernhout
http://www.pdfernhout.net/
https://narrafirma.com/automattic-javascript-engineer-application-by-paul-d-fernhout/
====
The biggest challenge of the 21st century is the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.


Replying to a Slack interview scheduling request from Nikolay Bachiyski

Date: 2015-12-22 (updated 2015-12-25 adding Seed Savers analogy and probiotics list, in part from reflecting on comments by another Automattician)

Nikolay-

Thanks for your invitation to text chat sometime the first week of 2016. Yes, I’m still interested. Any day from Tuesday 2016-01-05 onward would be OK with me right now for a text chat (we’re visiting my wife’s family before that). Timewise, any time from around 06:00 EST (13:00 EET) would be fine; later is better for me, but anytime after 6am my time is doable. A specific best time could be 14:00 EET presumably your time (7:00 EST my time) on Wednesday 2015-01-06, but I’m open to others (including if you’re traveling outside of Bulgaria). I’m not that comfortable with signing up for a Slack account right now for some reasons referenced below. Would IRC chat or Skype (as I already have an account) or a WordPress-hosted chat be acceptable alternatives to Slack? Or would there be some other option you could suggest? I’d also be happy to setup a server with SSL specific to the chat with some open source technology I’m working on.

To explain my request to use an alternative to Slack, I’ve added some details on my concerns about Slack for for both me and for Automattic at the beginning of my Automattic application page below.

–Paul Fernhout

Why Slack is problematical for me and Automattic

Some things to reflect on at the start:

  • What would Automattic do if Slack purchased SquareSpace, Klatcher, Sitecore, Acquia, or Wix to expand its offerings to web publishing? Or just did the same in-house?
  • What would Automattic do if Slack was purchased by Facebook or Google, who then integrated it into their other services (such as Pages or Blogger)?
  • What would Automattic do if a rogue Slack employee started leaking Automattic’s internal communications to the public? Or provide select information to other parties who paid for access? Has Automattic vetted all Slack employees for commitment to Automattic’s purpose of making the web a better place?
  • What would Automattic do if a third-party got access to Slack’s AWS credentials and started injecting occasional harmful words or code into Automattic’s internal discussions? Or even if third-parties just make lesser breaches and Slack does not even tell users the full extent of the breach?
  • What would Automattic do if Slack announced all of Automattic’s communications data had been deleted by “mistake” and Slack’s own internal backup procedures had failed? Is everything Automattic has in Slack backed up every hour of every day by Automattic? Even with backups, what would Automattic do at that point, go back to using Slack? If not, what could be done with those backups?
  • What would Automattic do if Slack started disallowing Slack users in some countries Automattic had employees in, or otherwise limited IP ranges for whatever reasons? Or what would Automattic do if Slack was banned or limited in Europe or India over some data retention or privacy issue?
  • What would Automattic do if Slack decided to inject annoying or potentially-malware-laden third-party advertising onto each Slack page?
  • What would Automattic do if users and WordPress plugin developers decided WordPress was out of style because it did not offer real-time instant messaging in the core like Slack and so could not support some nifty new feature most groups decided was essential for collaboration through the web after seeing it on Slack?
  • What would Automattic do if Slack changed its Terms of Service (TOS) to prohibit people from using Slack from working on instant messaging? Very unlikely, but look at the git/BitKeeper history.
  • What would Automattic do if Slack used Automattic’s internal communications for business intelligence to disrupt Automattic in some way, whether to poach key Automattic employees or launch competitive plans using ideas Automattic was discussing internally? Would Automattic even know that had happened? And does the Slack TOS actually prohibit that?
  • In general, what if the Slack TOS is subtly changed by Slack in the future in some way that is not good for Automattic and Automattic does not notice instantly and discontinue its Slack use before the change comes into affect, assuming there is even a grace period? What would Automattic do even then, if it objected to a sudden TOS change and both its internal and external communications are dependent on Slack?

My reason for that request to avoid Slack is that a major interest of mine is improving fine-grained real-time collaboration over the internet, and that is something I’d like to do at Automattic if given the chance; Slack is becoming a major proprietary centralized competitor in that space (with equity funding so far of about US$340M, or almost double that of Automattic’s $190.6M according to Crunchbase), and signing up with Slack could potentially create various issues down the road.

The FOSS NarraFirma WordPress plugin my wife and I developed is one such example of near-real-time communications tool, supporting multiple users working through a complex task together. For another example, given the ongoing crisis in the Mozilla Thunderbird project, my current folly is working on a FOSS proof-of-concept project for the Thunderbird planning list to consider for future directions for Thunderbird as a web server plus webapp (perhaps similar in deployment to Calypso, but also independently web-hostable) — and which perhaps someday might be considered a Slack alternative that is FOSS. Not much to see at this next link at the moment, but my ongoing work on that proof of concept is there (chat is next on the agenda after RSS feed reading and data storage):
https://github.com/pdfernhout/Twirlip2

So, I’m not too keen to get an account with Slack and agree to their terms of service if I don’t have to, because that opens up a potential legal can of worms for me. Even if their (lengthy) TOS and privacy policy were OK now, Slack reserves the right to change them as they want in the future. Still, I would get such a Slack account if I joined Automattic and so had to, because then the practical value would outweigh hypothetical risks. Plus I probably would not have much time for such follies, unless they became what I did at Automattic, and then we could talk about what made sense for the company not just myself. :-)

Statistically, at best there’s only a 20% chance of getting hired at Automattic starting with an interview and subsequent trial, given what I’ve read of Automattic hiring statistics on Dave Martin’s blog. So, I am reluctant to get a Slack account on speculation, given the potential legal or other downsides inn the 80% chance I won’t get hired. I also plan on applying to some open source companies like Kolab, Lingore where having a Slack account could perhaps count against me — although probably not at some places I might consider as well like Nylas, who have also picked up Slack sadly.

That percentage even ignores how I’m probably at the edge of hireable at Automattic and so the percentage of success is probably lower than 20% given I’m prone to writing stuff like this: :-)
http://ma.tt/2015/12/moral-responsibility-of-technologists/#comment-585177

Or, admittedly, worse, embracing Microsoft’s TypeScript while avoiding Facebook’s React:

Replace React with Mithril [in Calypso] for licensing reasons

Calypso could migrate to TypeScript

Because Automattic and other companies are using Slack and recommending it, it may be only a matter of time until I end up getting a Slack account through more general peer pressure though. But I’m hoping that day is not anytime soon. This is just the first time I’ve had to really think about that. I’ve been kind of dreading you asking me to do that and having to write and email like this as a response, but I kind of expected it. :-)

That is perhaps another reason I’ve been working towards something that could perhaps just as easily become a WordPress plugin we could use for conversing instead of becoming a possibly-Electron-packaged Thunderbird-ish desktop app . :-) Perhaps in two weeks (that first full week of January) we could use that Twirlip app on a server I hosted (or with IRC), with support for you downloading a copy of everything for your records — even if you were just going to paste it into Slack? :-) That would give me a lot of motivation to get that software working well (either for IRC or for self-hosted chat), so you’d be doing me a big favor by considering that possibility. :-) And if I can’t get a basic two-user chat server or IRC client to work well under Node.js in a couple of weeks, you probably should not be hiring me anyway — at least not as either a programmer or project manager. :-)

Although as a fallback, here is a Node.js demo chat system I wrote a couple of years ago in a few hours mostly just as an experiment and test for a data storage system — but I’d make something better with SSL and a fancy GUI for us. :-) I just sent you a chat message there, BTW, just to see if it is still working as I have not tried it in about a year (press “Refresh” to see it). :-) That site is not very private though, not SSL, not very user-friendly, not scalable, and has some other issues (including no authentication/authorization for user IDs), so we need better. :-) It’s mostly up there for fun and to see how people break it or if anyone pays attention to it. But it is at least a “proof of concept” that we can chat without Slack. :-)
http://twirlip.com/pointrel/pointrel-app/bootstrap/chat.html

It’s been sad to see Automattic move from IRC more towards Slack, given Slack is a centralized proprietary product. Slack also could be seen as a major Automattic competitor who can have access to much of Automattic’s internal real-time communications (whether appropriately or not). Slack is also putting those communications on third-party sites (potentially moveable whenever Slack wants, although AWS is hard to beat as a backend and is what Slack seems to use now). I know Automattic probably considers its competitors more as companies like Wix or SquareSpace, but I’d suggest Slack is an upcoming competitor too. It would probably not take that much to add basic web publishing to Slack. Rather than adopt Slack, I’d much rather see Automattic improve WordPress and WordPress.com (as with a ping server as described in my application) in order to help everyone do the things Slack supports either through WordPress.com or via their own WordPress.org sites. Slack does not give people that choice and I doubt they ever will — which is one reason why I am applying to Automattic and not Slack. :-)

Something else to consider: “Slack launches new $80M fund, app directory for platform partners

That is about half of the recent investment in Automattic that Slack is going to give out, just for a company that is coming from essentially nowhere, with no reputation of trustworthiness (not saying they aren’t, just not a record). It is hard for me to see how Automattic can completely concede that real-time messaging space given all the potential great WordPress plugins one can imagine in that space. But if Automattic partners tightly with Slack (maybe even getting a cut of such funds), then it is potentially letting all its open source messaging become dependent on a proprietary service.

None of this should imply I think Slack is not a useable product. It seems quite good from what I’ve read about it (although US$340 million can buy a lot of hype, astroturfing, and/or even outright lies). It is kind of amazing in all the discussion I read about Slack I have seen essentially nobody talking about the kinds of concerns I’ve listed here (although some only apply to a company like Automattic active in making the web a better place). So, I’m either completely off-base or US$340 million in equity funding can buy a lot of hype and astroturfing? Although now that I think to search on this phrase, I see at least one match (but not many):
https://www.google.com/search?q=why+i+would+never+use+slack

Here is the top match, which also mentions some new alternatives like irccloud.com, but even this one does not talk about privacy or centralization: “Please don’t use Slack for FOSS projects

“I’ve noticed that more and more projects are using things like Slack as the chat medium for their open source projects. In the past couple of days alone, I’ve been directed to Slack for Babel and Bootstrap. I’d like to try and curb this phenomenon before it takes off any more. … Slack is not a tool built for open source projects to use for communication with their userbase. It’s a tool built for teams and it is ill-suited to this use-case. In fact, Slack has gone on record as saying that it cannot support this sort of use-case: “it’s great that people are putting Slack to good use” but unfortunately “these communities are not something we have the capacity to support given the growth in our existing business.” … I often hear that IRC is dead. Even my dad pokes fun at me for using a 30 year old protocol, but not after I pointed out that he still uses HTTP. Despite the usual shtick from the valley, old is not necessarily a synonym for bad. … In short, I’d really appreciate it if we all quit using Slack like this. It’s not appropriate for FOSS projects. I would much rather join your channel with the client I already have running. That way, I’m more likely to stick around after I get help with whatever issue I came to you for, and contribute back by helping others as I idle in your channel until the end of time. On Slack, I leave as soon as I’m done getting help because tabs in my browser are precious real estate.”

The end of that article links to this thenextweb article on “Slack is quietly unintentionally killing IRC“:

“It’s not just communities using the platform, either. WordPress.org, the community behind WordPress’ open-source software, recently abandoned its IRC channel and shifted to Slack, citing that IRC is complicated and unfriendly. That shouldn’t be surprising to anyone who’s gone through the experience of using IRC for the first time; the barrier to entry was a formidable challenge for the first time user. To WordPress.org, it appears that Slack presented a way to remove that barrier and get more people involved. At time of writing there were over 3,000 users registered on its public Slack team. Over time, I expect more stories like WordPress’ to emerge, but perhaps that’ll only happen when Slack truly embraces what’s going on under its nose. One thing that’s encouraged communities to jump onboard with Slack is that it’s entirely free, as long as you don’t care too much about message retention. Considering how good Slack’s user experience is, this is a trade-off many are willing to make. It also works with IRC clients, for those that can’t quite pry themselves away from the old-school system. It’s a win-win for everyone.”

Doesn’t it bother anyone at Automattic that WordPress.org is now the poster child for centralized proprietary messaging? :-(

Still, it’s true that Slack apparently addresses some important needs compared to Skype that real companies face all the time (like central management, improved search, even bosses logging all employees’ communications, and some other things based on searching the web on “Skype vs. Slack”) even if Slack may have some of its own issues (including for some the bosses point, although I assume essentially everything online is logged by someone). I myself use centralized sites, and a proprietary Skype and Twitter and so on, which make me not a “purist”, even as I’m pushing back some on Slack in particular at the moment. Slack may well be a terrific short-term business decision for a lot of companies. Of course, one could have said that about Moveable Type in 2003 too. :-)

Here is another precedent as well, from the history of git: “The kernel and BitKeeper part ways

“The only problem was that BitKeeper is proprietary software. Instead, it came (in binary-only form) with a license which allowed free use, but which imposed some significant restrictions. The free version of BitKeeper could only be used with open source projects; users could be required to make their repositories available on demand. The free version posted all changelog information on openlogging.org, and disabling the logging was not allowed. Users were required to upgrade to new versions, which could come with different licenses. And users were not only prohibited from reverse engineering the software, but they were prohibited from working on any sort of source code management system at all.”

See that last item is why TOS can be worrisome.

People got used to putting confidential and strategic information into centralized Google docs. Now they are getting used to doing that with Slack. I’m reminded of this Milton Mayer excerpt about the loss of freedom in another time and place, which quotes someone else:

“To live in this process is absolutely not to be able to notice it—please try to believe me—unless one has a much greater degree of political awareness, acuity, than most of us had ever had occasion to develop. Each step was so small, so inconsequential, so well explained or, on occasion, ‘regretted,’ that, unless one were detached from the whole process from the beginning, unless one understood what the whole thing was in principle, what all these ‘little measures’ … must some day lead to, one no more saw it developing from day to day than a farmer in his field sees the corn growing. One day it is over his head.

How is this to be avoided, among ordinary men, even highly educated ordinary men? Frankly, I do not know. I do not see, even now. Many, many times since it all happened I have pondered that pair of great maxims, Principiis obsta and Finem respice—‘Resist the beginnings’ and ‘Consider the end.’ But one must foresee the end in order to resist, or even see, the beginnings. One must foresee the end clearly and certainly and how is this to be done, by ordinary men or even by extraordinary men? Things might have. And everyone counts on that might.”

IMHO, Slack centralizing communications under proprietary software and destroying IRC is a problem Automattic should be addressing, not a problem Automattic should be facilitating. :-( But I already get the feeling this is a lost battle. :-(

My wife is *really* going to be cross with me if I tell her things fizzled over just a Slack account. :-(

And, realistically, even if Nikolay said yes to some chat alternative, then subsequent interviews would require a Slack account, and the trial would require a Slack account, and the Matt chat too. Sigh. So, it is quite reasonable for Nikolay to say at this point things won’t work until that internal dynamic changes (if ever). And I can respect that — organizations are what they are, with their own cultures and dynamics. I got interested when Automattic used IRC, but things change… I just may be too old-school (local-oriented, decentralized-admiring, privacy-option-creating, and even too much of a worry wort on too of that) and thus too out-of-sync with Automattic’s current directions to make it work. I can even wonder if, sadly, it may just be easier contracting somewhere like NBCUniversal where I just expect many things to be messed up (both technically and regarding the mission) and then am pleasantly surprised occasionally when they are not — rather than vice versa. At least as a contractor, psychologically I can remain at arms-length from the internal issues.

I know nobody is perfect though (least of all me). I know Automattic is doing the best job it thinks it can collectively with limited resources. The choice of React I can understand even I have issues with it as in the issue I reported on GitHub — I almost used React myself, given the community. I would not have a big problem coding in React myself given it’s been chosen for Calypso (especially given broad community support), even if I feel Mithril is better technically. But Automattic’s choice of Slack I just can’t understand (even if I can agree IRC has issues — but Automattic could have fixed them itself). I guess my vision of what Automattic can or should do is just broader than what it is now — although I can respect all organizations have limited resources and need to pick their battles. But it is very sad to me Automattic decided not to pick the battle of fighting for real-time collaboration using FOSS software for WordPress users. :-(

Writing this essay broke WordPress once, but together with the community, we were able to fix it (especially thanks to @adamsilverstein and @miqrogroove). I’m continuing to edit this application with the patch Adam Silvestein wrote in place (putting it in via a git stash pop and pulling it out when I’m done, in case of auto-updates). But fixing WordPress through a few coding changes is much easier than fixing an essentially infectious proprietary disease like Slack, a proprietary disease backed by US$340 million of venture capital (and likely soon more).

The situtation me of some zombie or vampire stories, like Phineas and Ferb and the Night of the Living Pharmacists? “I want you to be Slack just like me…” :-) Sad. :-(

Medically, to deal with a disease outbreak after it gets going, one uses things like quarantine, improved sanitation, probiotics, antibiotics, immune system boosters, immunization, improved nutrition, and reduced health stress (among other things like disease management planning and coordinated communications and so on). If you are Phineas and Furb, you figure out you can just douse the Zombie Pharmacists with water — but you have to do it all at once. :-)

I don’t really know yet the equivalents for those for this heavily-funded Slack infection that appears to be about to consume the world and perhaps destroy IRC and maybe tear apart many FOSS communities in the process. As Phineas and Furb said, if you don’t get all the Zombie Pharmacists at once, they will just spread the disease again. And the challenge is potentially even worse with an infection which can in theory monitor and sabotage discussions in real-time about creating alternatives to it.

But one equivalent may be “probiotics” of free software, which help crowd out the Slack infection. Several Slack alternatives were suggested in this Hacker News article. such as Mattermost, Tox, Matrix.org, Zulip, Rocket.Chat, Let’s Chat, and more. There is also what I’ve long been working on with Pointrel/Twirlip. And there is also Kolab. Here is a list of open source software for collaboration collected at Wikipedia. Still, that does not mean they can all meet the needs Slack meets. Nor are they as well known as Slack, given Slack has US$340 million of investments it can mostly pump into advertising and customer relations. I can still wonder if something a lot better might make a difference. As one insightful exchange on that Hacker News article went:

 “This is more like “don’t use BitKeeper, use git”.

  “No, it’s more like “don’t use BitKeeper, use CVS.”

And as another comment there put it:

“There is a chance to repeat git success story. Remember what happened?

1. Everyone uses cvs or svn.

2. Bitkeeper comes along with a good DVCS, gets adopted by Linux.

3. Linus rants about closedness of BK, reimplements it as git.

4. Git takes life on its own gets wildly popular and makes way for many awesome things such as github and gitlab.

Replace svn by irc and bitkeeper with slack. Figure out the rest.”

That “rest” could have been based on WordPress, but Automattic seems to be throwing away that opportunity.

Still, people can co-evolve with infections. For example, T. gondii may be in play right now in human evolution, related to cats who carry it and are beneficial at keeping down mice, while T. gondii itself may contribute otherwise to some mental illness. There can also be helpful infections as well as  harmful ones, given 99% of bacteria are helpful or neutral and humans need a healthy microbiome to survive. Most people can get a flu or cold and survive. I can hope the web community as well as Automattic can co-evolve with the Slack infection eventually.

I can hope these risks never play out for Automattic or FOSS communities. I may even feel forced to get a Slack account eventually just so I can pay the bills and feed my family and so we don’t lose our home after so much time spent writing FOSS like NarraFirma and now Twirlip (and probably by that point, working for a place I’d less like to work for than Automattic all things considered, with a long commute and probably on proprietary software).

But as a FOSS developer, I don’t have to like that situation. I don’t have to like Slack, no matter how conveniently that centralized service works. And what I most don’t have to like is that Automattic has missed a tremendous growth opportunity for FOSS software development. :-( And I don’t have to like that Automattic has promoted centralized proprietary software as the way for free software developers working on WordPress should interact.

As an analogy, consider Seed Saver’s Exchange whose mission is “Saving America’s Heirloom Seeds”. Such seeds are free and open source in the sense that they are not covered by copyrights or patents, and you can save your seeds from this year’s crop and grow from them the next year’s crop (which you can’t legally do with a lot of commercial seeds now). As they say in their story, “Diane and Kent went on to form a network of gardeners interested in preserving heirloom varieties and sharing seeds. Today, with 13,000 members and 20,000 plant varieties, Seed Savers Exchange makes its home on 890 scenic acres in Winneshiek County, Iowa, at Heritage Farm.” Now, imagine if Seed Savers decided to “improve” those acres to make them more “open” to stakeholders who were coming for a big get together to create the future of Seed Savers. Imagine, to spruce up the place, Seed Savers replaced all the flowers by the main buildings with commercial proprietary flower varieties because they were easier for gardeners to get and cheaper to buy. Imagine they served only GMO produce from Monsanto at the get-together because it was likewise cheaper and easier to get and such food had some advantages claimed by Monsanto (a major developer of proprietary GMO seeds). Imagine they even announced a big partnership with Monsanto at the get together and on the CEO of the organization’s blog. And, imagine Seed Savers paraded these decisions in front of stakeholders of evidence of how hip and cool and “open” and heirloom-gardener-friendly they were being, and said that using Monsanto products at the heart of their operation did not matter to their mission because they were still growing heirloom seeds in the actual fields. And imagine people came along to suggest trying new varieties of free and open heirloom seeds to meet whatever needs Monsanto was supposedly meeting and just got told there was not enough time or money to consider that. How many stakeholders do you think would walk away from that stakeholder meeting shaking their heads in bewilderment? Or worse?

This comment by “Gordon” to “Slack, Atlassian’s HipChat race to replace office email” seem well worth pondering both regarding email and also regarding creating WordPress plugins for instant messaging, email clients, email gateways, email mailing lists, and collective archives:

I don’t need new revolutionary replacements for email, I just want email to work. Please web standard boffins, please start putting together Email 2.0 protocols. I want standard encryption protocols, and Key-Servers added to the DNS protocol. I want email to to handle VOIP, Video and messaging without signing up to third party systems. I want to know that my email 2.0 features will work across a distributed network of many providers, none controlling or owning the technology. I don’t care that E-Mail is old and daggy, or that it comes from a time when the letter ‘E’ was the rage. We are fickle users, and we flirt with new things, but E-Mail (Can we still call you that?) will always be our first love.

BTW, Nikolay, whether we work directly together or not, and it’s looking more like not, if you’ve got three words I should keep in mind when writing FOSS chat or other communications software, I’d love to read them. :-)

(Feel free to post or link to this note for others at Automattic to consider as well if you want if you are still having internal discussions on moving further forward with Slack versus alternatives.)

–Paul Fernhout
http://www.pdfernhout.net/
====
The biggest challenge of the 21st century is the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.


Replying to first contact from the WordPress Security Czar!!!

Date: 2015-10-14

To: Nikolay Bachiyski <…@automattic.com>

Hello Nikolay,

Thank you for considering my application and planning an interview in a few weeks. It’s exciting to see my chances of working at Automattic increase from perhaps 1 in 1000 to perhaps 1 in 5 (according to Dave Martin’s statistics and other sources). As you asked about whether the timing works for me, certainly sooner rather than later for an interview might be better, given I’m currently job hunting. With only at best a 20% chance (statistically) of successfully moving through the rest of the hiring process with Automattic from this point onward, it would be difficult (but maybe not impossible) to convince my wife that I should pass up a sure thing — even as I’ve been intentionally dragging my feet on sending out many other applications for that reason. :-) With that said, no doubt there are many amazing applicants to consider, all with great ideas, great skills, and great past experiences. So, try as I might, I can’t come up with a good reason to suggest you schedule my Automattic interview any quicker than the rest. Automattic has a current process, with people doing the best they can can given a big challenge, and I can respect that — even if, admittedly, I’ve tried to circumvent that process a bit before when the odds were 1 in 1000. :-)

Looking forward to something that  is hoped to be fun can also be a great source of happiness, whatever the ultimate outcome. :-)

So, I’ve upgraded my WordPress.org profile status from:

“Software Developer at my own sole proprietorship, but currently waiting to see if jobs@automattic.com can ever catch up on their email backlog… :-)”

to:

“Software Developer at my own sole proprietorship, but currently waiting patiently in the Automattic interviews queue… :-)”

Given your current role as WordPress Security Czar, I’ve added some related rambles on security near the top of my application page, and I hope we can someday chat about them. In particular, I added a new link in those rambles to a document you might find of interest related to cyber-security. It was an (unsuccessful) proposal my wife and I sent to DARPA in 2011 on collecting and organizing cyber-security stories so everyone globally could learn from them and get better at securing systems everywhere. I also suggested there how that proposal could relate to issues you raise in your WordCamp Europe 2015 talk on WordPress security principles. Undoubtedly you have a lot to do already as WordPress Security Czar, but I can hope some of the ideas in that DARPA proposal might be worthwhile for you and others at Automattic to consider someday as part of promoting security consciousness within the WordPress community; I would be happy to see a related project get going, whoever was lucky enough to work on it.

All the best.

–Paul Fernhout
https://narrafirma.com/automattic-javascript-engineer-application-by-paul-d-fernhout/


Rambles on security issues for Nikolay Bachiyski or others

BTW, Nikolay, congratulations on becoming the new WordPress Security Czar. Or maybe I should write “commiserations” — as dealing with things like exploits possible from combining multiple weaknesses as you mentioned in your interview (referencing this link) has got to be challenging? :-)

Best practices as you reblogged about mentioning Henrik Warne’s essay can help improve software quality and so reduce likely vulnerabilities, no doubt. And they are good to do for many other reasons, as well.

Another document along those lines by someone else is “A Conceptual Framework for System Fault Tolerance” from 1992 by Walter Heimerdinger and Charles B. Weinstock. While it is not specific to security, if you view a security issue as a “fault” or “failure” then the terminology and conceptual background there may be helpful if it is unfamiliar (although it relates to things you’ve mentioned elsewhere about security and layers). It could be interesting to go point-by-point through Henrik Warne’s list and see how it maps onto the CMU Software Engineering Institute document.

Along more specific lines to enhancing global cyber-security, in October 2011, I emailed a bit with Peter G. Neumann (of RISKS Digest) about perhaps using some software (like my wife’s Rakontu system or similar) to collect real-life cyber-security stories — based on an (unsuccessful) grant proposal we sent to the USA Defense Advanced Research Projects Agency (DARPA) — the same people who helped fund part of the beginning of the internet. Peter Neumann actually liked the idea and suggested I post it to RISKS Digest, but that was just before I started contracting at NBCUniversal, so the idea got stalled on my side with my becoming busy with a new job. I’m just reminded of it by your contact now.

In case you are interested in it, I just put a slightly redacted copy of that proposal on my website at this link: “Participatory Sensemaking about Real-Life Cyber-Security Stories with Rakontu 2.0“. While that proposal is slightly out-of-date because we have moved on from Rakontu to JavaScript-powered NarraFirma as a possible platform for future developments, and was a bit ambitious in what we could do in a short time, you might want to put similar ideas into practice in the WordPress security community in your own way. Not that you probably have time for even more work, of course. :-)

From the executive summary of that DARPA proposal:

“Cyber-security professionals need examples of security successes and failures to make inferences about threats and opportunities. Examples support the emergence of broad strategies and collective values that support critical decision making in times of need. Collective structured dialogue and analysis tools applied to thousands of real-life non-fiction cyber-security example stories (such as in RISKS Digest) could help professionals make sense of complex security issues. We would like to create essentially an advanced distributed RISKS Digest-inspired system. To do this we would like to improve our open-source social media platform (Rakontu) to more fully support Participatory Narrative Inquiry, a sensemaking approach based on stories of real-life experience. We would like to do this in an agile way with the feedback of cyber-security professionals (both novice and experienced) who gain real benefits from using Rakontu for collective sensemaking.”

Thank you for your presentation on security at WordCamp Europe 2015: “Nikolay Bachiyski: A Few WordPress Security Principles“. As an example to connect your presentation to that unsuccessful DARPA proposal, consider the stories you mention about WAPUUIFY or the school website forums in your talk — stories used to illustrate key ideas like “security as a process”. Such stories could perhaps be added eventually to thousands more stories in a shared repository (a distributed one perhaps, perhaps with local private stories added too). Such stories as you mentioned could be added along with more stories illustrating other ideas you mention like thinking playfully like an attacker and that the security of a systems is the same as that of its weakest link. More and more such stories could be added over time by various people. By looking through such stories, everyone could learn from them as-is, or, even better, people could use the collection to “distill” more cyber-security best practices, perhaps through face-to-face workshops at conferences facilitated using various participatory sensemaking workshop techniques.

While reading such stories might not help people learn much specific technically, as you point out every developer needs to learn these concepts to apply to their own projects continuously. Such a collection of stories might help with that. I know I’ve learned a lot by reading RISKS Digest in the past, as well as other stories such as on Slashdot. However, such stories generally are found here and there, and so are not comprehensively searchable or visualizable as with graphs to look for commonalities or trends.

Ultimately, being a good programmer is probably not so much about knowing the best ways to do something compared to being able to recognize hundreds of ways of how not to do things (generally after you do them yourself) and then learning how to recover from such situations more and more quickly. Those issues range from typographical errors, to incorrect implementations of algorithms, to inefficient architecture, and all the way to mistaken requirements. While many of those good practices are just learned by direct experience, it is usually cheaper to learn from someone else’s experiences. Mithril, for example, reflects Leo Horie’s broad experience with website creation, so such experiences can also get encoded into libraries.

Even with benefits of learning from others, there is still great benefit to learning by doing though including to motivate further reading. So such a collection of security stories is not a replacement for other approaches to learning about cyber-security or other tools for doing comprehensive audits, automatically checking code, performing regular tests, and so on. Such a story collection would just be an extra supplement to such other activities. However, such a supplement of lots of security stories might fit well with the ethos of the WordPress community, which likes to write. :-)

As you say in your talk, no one is perfect. The NarraFirma plugin itself may still have some issues. For example, even though we used Mithril which is safely escapes all the user-supplied content in generated HTML by default to protect against various XSS attacks and such, there are some places we felt we had had to display otherwise-generated HTML, and I could well have made a mistake there. Also, I’m not that experienced a PHP programmer specifically (while otherwise very experienced generally), so I might well have made some rookie mistake there as well. That’s why on GitHub we still list the software as “pre-release”, although I did not see a comparable choice for WordPress.org — although we tried to signal that by picking a version number less than 1.0.0. To be clear, I know of no such vulnerabilities specifically in NarraFirma; I’m just being cautious.

The approach I’ve taken with NarraFirma plugin was a simple core (~1% of the code) in PHP to define a message-based triplestore with the rest (~99%) as a complex application in JavaScript/TypeScript. That approach to plugin development might help enhance WordPress plugin security if broadly adopted. While such an approach might not protect against XSS attacks (since the JavaScript may be poorly written and not use inherently safer libraries like Mithril), such an approach can at least help reduce the overall attack surface by reducing the amount of PHP code from many plugins that interacts with the WordPress core and so might do unexpected things. Almost all of the NarraFirma plugin could be completely reusable for thousands of applications that take the same approach, and so could perhaps become its own standard “messaging triplestore” plugin providing such a service to multiple other plugins (or maybe even make it into the WordPress core perhaps somehow if it proves generally useful). If JavaScript-based WordPress applications often used a common core service like that, that could make plugin security review somewhat easier — even if XSS and similar issues possible with bad JavaScript would still remain a risk. The advantage to such an approach is that, when (not if) sites get compromised from insecure plugin programming, cleaning up afterwards could be a lot easier, since the main part of WordPress and the database would not be as greatly at risk (beyond deleting some bad data). Almost everything I’ve read suggests attackers will succeed eventually; so while you want to minimize that obviously, you want to be extra sure to make recovery both possible and easy. Granted, the sorts of plugins that more readily fit in to such a model of using a common backend for basic storage tend to be more “decoupled” compared to many existing plugins, so that approach does not work for all plugins (like, say, those creating shortcodes), just a subset of plugins. But every little bit helps. :-)

My personal belief is that security is ideally best based on ideas about “intrinsic security” like Amory Lovins focuses on (as in, “Our tough systems are resilient under stress”), and Morton Deutsch’s ideas about “mutual security” (as in, “We are all in this together and so we are all watching each other’s backs”). That approach is generally a far superior strategy compared to creating systems relying on extrinsic security (“Our brittle systems and long supply lines are secure because we pay for a lot of guards”) or otherwise creating systems that are based on unilateral security (as in, “I somehow feel secure despite my brittle systems because I know I make everyone else around me nervous which deters attacks…”).

As mentioned in that linked DARPA proposal:

“In the very long term, people sharing stories about defense and abundance may lead to a new vision of intrinsic security and mutual security that reduces the overall level of strife on the internet.”

However, I obviously have not (yet :-) managed to persuade DARPA to redirect some of the about one trillion US dollars a year the USA spends on “defense” towards such a strategy of intrinsic mutual security — even as I can hope that just submitting such a proposal may have some beneficial effect on DARPA one reviewer at a time. :-)

Unfortunately for my home country of the USA, failure to prioritize such cyber-security issues sooner (either as my wife and I proposed or in lots of other possible ways) lead to the biggest significant data breach in US history at the US Office of Personnel Management (OPM). That breach compromised the personal information of about 20 million US citizens who have worked with (or in some cases applied for work with) the US government — including probably the detailed background information on many (or perhaps all) people with US security clearances. Frankly, it is hard to imagine a worse data breach than that as regards US national security (excepting obviously something like a breach of nuclear missile security or maybe some critical infrastructure breaches). While the movie “WarGames” at that previous link is full of laughable computer situations (even when it came out), the line at the end remains insightful, that sometimes “the only winning move is not to play”. Unfortunately, I’d expect some big part of the USA security apparatus still thinks it can “win” at cyber-security by aggressive moves like creating Stuxnet, weakening encryption, putting in backdoors everywhere, and so on. To me, and many other people, that all seems like a prescription for intrinsically insecure systems which are then vulnerable to all sorts of attacks from anywhere. In a global society that is increasingly dependent on computers (for good or bad), that just seems like a recipe for global disaster.

Unfortunately, sadly, my own country has apparently repeatedly demonstrated (as above) an almost complete lack of institutional skill at fostering true security for its own people let other people around the globe (even as the US military can certainly make a lot of noise and worse). Given that obvious “void” in true security skills, perhaps, say, a NATO member like Bulgaria might someday lead the way on such strategic directions towards true security, just like Finland led the way on Linux, and so provide a good example for the rest of the world to follow — including, maybe someday, the USA as well? :-) Or, obviously, a global organization like Automattic could provide some of that leadership too. :-)

As you said, at the end of your WordCamp talk in response to a question about online tools related to security issues, if anyone else has any ideas, “feel free to share”. :-) So, I’m sharing. :-) I hope you find some of this useful, whether we end up working together or not.

–Paul Fernhout
http://www.pdfernhout.net/
====
The biggest challenge of the 21st century is the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.


Breaking (and then improving) WordPress

Update: WordPress Trac issue #34560 “High memory usage (and possible server error) editing post with many/large revisions” is related to this essay. I reported that issue based on server errors and white screens I experienced at the end of editing this long essay as it grew and grew (until I could not add to it anymore from such errors). Obviously, WordPress must have been trying to tell me something. :-) I included a suggestion for a possible fix in the issue report, but the suggestion turned out to be incomplete as I only had noticed one of three causes of the high memory use issue. The actual complete fix in PHP was done by Adam Silverstein (with suggestions by Robert Chapin) after a few iterations of testing and feedback. As I wrote there:

@adamsilverstein Thanks for the great patch, and @miqrogroove thanks for helping with it too. I am testing it now with the production server with the issue. Previously, the editor failed to open a ~437K page due to a server error from exceeding 256MB. Now the editor page opens, and memory use according to WP-Memory-Usage is 27.96MB of 256MB. So, huge improvement! As a baseline, 23.68MB of 256MB is used in the admin plugin panel on that server, and 26.55MB of 256MB is used by the editor when creating a new page. So, only about 1.41MB is being used by the editor above the new page baseline for the really big page. Wow!!! … So, it seems that WordPress was indeed loading all the revisions of a post *three* times when editing it with the edit_form_advanced: once to count the revisions, once to list the timestamps and other metadata of the revisions, and once to check for the latest autosaved revision (if any).

And as I also wrote there: :-)

It’s a tribute to how much people love WordPress that this inefficiency has been lived with for so long, since this peak resource demand may increase routine server costs (like with a VPS often chosen by maximum memory) to avoid server errors when editing in edge cases. :-) Either that, or people like me who would try to write an overly-long overly-verbose under-edited under-proofread rambling book-length 64000-plus-word autobiographical essay, that essentially no one is ever going to want to read, as a single WordPress page with lots of revisions (instead of taking some other more sensible approach to effective writing) are thankfully rare. :-) Or maybe both? :-)

This essay spawned three other “Meta Trac” issues I reported as well based on my experience reporting the first issue:

  • #1378 “Confusing wording when creating new ticket in WordPress Trac” (fixed within an hour by teamwork between Dion Hulse and Andrew Nacin, wow!)
  • #1379 “Lost text without warning when click link in Trac without JavaScript enabled” (a relatively minor issue that probably won’t affect most people)
  • #1381 “Trac could have a “Thank You” button or link” (a long-term speculative research idea)

Thanks to Adam Silverstein and Robert Chapin, I can now edit this essay again and even add more to it. :-) However, I’ll probably leave it as it is at this point, despite lots of possible further improvements, as I need to move onto other things. It was an interesting experiment though in pushing a single WordPress page to some technical limit and beyond.

My thanks also go to Tom Nowell (who I contacted in part to thank for his essay) for kindly suggesting how my essay could use a table of contents which I then added via a plugin. He made several other helpful suggestions too, although sadly I don’t have time right now to implement them. Maybe someday, perhaps as part of breaking something else about WordPress? :-)

Whether I end up working for Automattic or not, I’m glad writing this long essay has already (indirectly) helped improve WordPress and WordPress.org including (when Adam Silverstein’s changes make it into core) by reducing the number of frustrating-to-debug server errors other WordPress users will see. :-)

–Paul Fernhout

(My mention of any people’s names here is not meant to imply the named person in any way endorses this application.)


Cover letter

To: jobs@automattic.com

Date: 2015-10-14 (last revised 2015-11-13)

Dear Automattic Team-

NarraFirma
JavaScript engines
dreaming triples on stories
shared through WordPress

Please accept my application for the “JavaScript Engineer” position at Automattic. My main interest in the role is to improve WordPress as an application platform supporting complex “decoupled” JavaScript applications using AJAX/REST with JSON (see links below). A good match might be for me to work with the Automattic Janitorial team (or perhaps the experimental products team). Based on the needs of other Automatticians, interpreted through my own experience and interests (like to support stigmeric and chaordic social processes and crowdsourced public intelligence and sensemaking), as an “Janitor”, I could implement a variety of low-latency multi-user FOSS JavaScript-powered WordPress plugins and supporting services as experiments (made possible by a “PingPong” WebSocket server mentioned later) to help the Automattic community be even more productive and enjoy their jobs even more. Some of those plugins would be “decision support tools” useful for public intelligence and sensemaking about the growth and health of Automattic and the larger WordPress community. Others might be “educational simulations” on key ideas of importance to Automatticians, like to help new hires more easily understand stigmergic or chaordic processes or other aspects of Automattic’s ethos. Still other plugins and related services might be intended to help with other concerns at Automattic in different ways including just as research experiments for further feedback and iteration, like for example making it easier to say “thank you” on Trac without disrupting someone’s workflow. As with Jetpack, those plugins and supporting services could then help strengthen the rest of the WordPress community as well, and they could also provide new opportunities for Automattic to profit by offering related server hosting.

The rest of this long document attempts to relate my life experiences to Automattic’s and the WordPress community’s likely current and future needs, based on my reading about them (including “The Year Without Pants” by Scott Berkun). It also outlines several projects I think might be worthwhile to do at Automattic. However, if instead you’d prefer a terser document with more specific dates and detailed lists of technologies I’ve worked with on various projects, you can find my current resume here[link removed 2016-01-05]. We can work out the rest 140 characters at a time. :-) So, only read further if you want. :-)

How did you learn about Automattic?

I first saw the Automattic JavaScript Engineer opening around March 2014 while setting up a WordPress site for my local historical society. I then read related discussions of working at Automattic and Automattic’s future plans. I thought, “Wow, I want to apply for a job at Automattic helping support and improve much of the web working with that great team!”

Why didn’t you apply to Automattic right away?

Given how (no doubt) everyone wants to work from home for Automattic on open-source software, I knew I’d need an impressive demo of WordPress-related software. As the JavaScript Engineer job description page says: “We are lucky to receive hundreds of applications for every position, so try to make your application stand out.” So, I figured I’d have to first create an amazing WordPress site demonstrating some JavaScript-heavy plugin I wrote to get your attention. And of course I would also have to learn to spell “Automattic”. :-)

[The scene is just after many costumed superhero wannabees audition at a pool party and are all rejected for failing to meet the Mystery Men’s expectations, and then the Bowler shows up with an unusual skill]
Mr. Furious: We’re an elite cadr-cadrey…
The Bowler: Cadre.
The Shoveller: You’re in.
(This and other movie quotes are from “Mystery Men“)

So when my wife and I decided to rewrite her story work software for the web, WordPress came immediately to mind. I’ve been working towards a hopefully-stand-out multi-user decoupled open-source JavaScript/TypeScript single-page web app called NarraFirma ever since.

What is NarraFirma?

NarraFirma is a WordPress plugin, and it is also Node.js hostable. It has more than forty virtual “pages” that together help a person move step-by-step through a complex “Participatory Narrative Inquiry” process involving data entry, graphing, statistics, clustering diagrams, recommendation making, survey taking, import/export, printable output, project configuration, progress dialogs, page-specific online help, and more. I have been working on it pretty much full-time since around August 2014 — or more than 3000 git commits ago. :-)

NarraFirma WordPress Admin Panel Screenshot

NarraFirma scatterplot screenshot

 

Now that a first version of NarraFirma is finally done, you can:

… as essentially my hopefully stand-out Automattic “application”. :-)

Why did you not just write NarraFirma as a plain PHP plugin?

In general, by writing mostly JavaScript we also kept our deployment options open (since NarraFirma can now run almost anywhere JavaScript can run). Also, I just like JavaScript better then PHP, even though all programming languages have strengths and weaknesses in different contexts.

For more about the “decoupled” approach to using WordPress as a back end to a complex JavaScript front end (and some other inspiration for why I chose that particular architectural approach for NarraFirma), see these links.
http://wptavern.com/decoupling-wordpress
https://pantheon.io/what-know-about-decoupled-cms-recording
http://wptavern.com/3-big-things-that-will-happen-to-wordpress-in-the-near-future
http://wptavern.com/in-the-next-few-years-90-of-wordpress-development-could-be-javascript-based

And see especially this next link, and this quote from it:
http://premium.wpmudev.org/blog/wordpress-rest-api/

“Anyone paying attention to Matt Mullenweg’s public appearances over the past twelve months will have noticed the steady beating of one particular drum: The importance of the WordPress JSON REST API to the future of the platform.”

So, in a way, I’m just following some good advice to focus on JSON and REST and such, even though NarraFirma may not have been exactly what Matt Mullenweg had in mind (but that’s the value of diversity and community and open source).

Why do you think you would be a good match for Automattic?

One of my best qualifications for a position at Automattic is that I like to write both text and code. I also like to read both text and code. The Automattic trials/auditions may be a success in part because they check for that combination (among other things like motivation and interaction style). People who can write well and code well (and who can read writing and source code well) tend to make the best software developers. That’s because software needs to be understandable by other people, and it needs to actually work. It also must fulfill requirements and connect to other systems; thus reading is essential to understanding the context of the application.

I hope that my ultimate success in getting NarraFirma out the door demonstrates a self-driven work ethic, as well as curiosity and the desire to learn. That success and my past FOSS projects also show a love of Open Source software — even when “love hurts” financially. :-)

I also have experience working from a home office for years, like the two and a half years I provided agile development services and third-line support full-time for NBCUniversal’s broadcast operations.

Is there anything else you want us to know up front?

Helping my wife related to narrative sensemaking, I’ve also worked on intelligence tools used by governments to hopefully help produce better decisions considered from a variety of perspectives. I would like to bring a lot of those ideas into the WordPress community to help empower the community (including at Automattic) to do better mixed-methods research and sensemaking about opportunities, risks, trends, and more, to help the WordPress platform and WordPress community become even grander than it is today.

Heller: I saw the action out there tonight. What you guys need is a little firepower.
Shoveler: Well…
Heller: I’m a weapons designer. (grabs Shoveler’s collar) I’ve got what you need.
Shoveler: A–All right. We’ll–we’ll come back. We’ll visit all of you people, later. Thank you, sir.
Heller: No no no no no. I–I don’t live here. I’m here for the ladies. You know. Here, take my card.

How can we find out more about you?

Below is more background on NarraFirma and myself, written at length (but with tl;dr summaries) in the spirit that communication is oxygen. Hopefully that background will show my ongoing commitment to free and open source software and content despite many obstacles (including my own fumbles). Interspersed with my history are several ideas for projects at Automattic thought of as I tried to translate my life experiences and aspirations into an Automattic/WordPress context (search on “Automattic” to quickly find them).

One example is setting up a proof-of-concept “PingPong” or “HeadsUp” new-data-available notification server at Automattic as a trial project. Such a server and related WordPress code could potentially make any JavasScript-enabled WordPress page update with low latency in near real time. That would create a new frontier for the Automattic/WordPress community to explore together like to replace IRC chat or to support team whiteboards.

Still, as my wife points out, this document is way too long and will likely preclude me being hired. She may well be right, although she has not read as much about Automattic/WordPress as I have. Even if she proves right, I hope some of the ideas in this document may prove useful to Automattic someday, especially as Automattic continues to grow and faces new challenges relating to coordination and sensemaking. In “Mystery Men”, the Shoveler thought Dr. Heller was crazy at first too, even if the team warmed to him in the end. :-)

How can we get in touch if we decide we’d like you to audition?

I hope we can find a way to make more great WordPress software together soon and expand WordPress’s successful approach into other Automattic services. But, even if not, thanks for the inspiration to write some more free and open source software! :-)

–Paul D. Fernhout
email: pdfernhout@kurtz-fernhout.com
Location: Edinburg, NY, USA (in the Adirondack Park)
Phone, Skype ID, and street address available on request via email
https://twitter.com/pdfernhout
https://github.com/pdfernhout
https://www.youtube.com/user/pdfernhout
http://pdfernhout.net/
====
The biggest challenge of the 21st century is the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.

Do you have any musical recommendations for listening to when reviewing this?

Well, there is a lot of great music out there. For example, I listened to the soundtrack of Mystery Men over and over when writing some of Post-Scarcity Princeton, and I also watched that movie once again during the middle of writing this essay, and kept that music in mind. Towards the beginning of the process of working on this essay (especially the NarraFirma sections), I listened a lot to Ivan Chew’s “I Have Often Told You Stories” which is used in the NarraFirma YouTube video. Towards the end of the process, I started listening to Earbits’ newgrass channel, such as for example “Nooks and Crannies” by Leftover Salmon, “Leaves and Pods” by Lindsay Lou & The Flatbellys (many times), and similar from that channel. For putting in all the subheadings at the end (which took several hours just by itself), as well as additional proofreading and edits, I listened quite a few times to Mandolin Orange’s 2010 New Year’s Eve concert in Raleigh, NC. I also listened to “Truckin’” and then many subsequent Grateful Dead songs on YouTube (all the way to “Terrapin Station” and beyond) when editing the Halloween/Porcupine section near the end. Of course, your musical tastes may differ and probably do. :-) But if you want the full “experience” then that is the kind of music that might go best with reading this essay.


More about NarraFirma


tl;dr NarraFirma is cool software because:

  • it matches Matt Mullenweg’s suggestion about the future of WordPress with JSON by using WordPress as an application platform
  • it implements my wife’s design for helping people make sense of their own life-experience stories
  • I put more than a person-year into writing it on top of neat libraries like Mithril and D3, while learning a lot about JavaScript and other web technologies

What does NarraFirma do?

In a nutshell, NarraFirma helps people gather stories from a community or organization and make sense of them to inform and improve collective decision making. The software supports project planning, story collection, data analysis, facilitation of group sessions, and recording of project results. More information about NarraFirma can be found on the main NarraFirma site, a Knight News Challenge entry on “How might we make data work for individuals and communities?” , and an announcement on my wife’s blog.

My wife did most of the high-level application GUI design for NarraFirma based on the step-by-step process outlined in her book (and based on nine previous pieces of related software, some of which I helped her implement). I designed NarraFirma’s supporting architecture and did most of the implementation.

Where did the NarraFirma idea come from?

The NarraFirma application’s requirements were mostly based on my wife’s (Cynthia Kurtz’s) Creative-Commons-licensed 700-page textbook, Working With Stories in your Community or Organization: Participatory Narrative Inquiry. NarraFirma is about the tenth tool Cynthia has designed to support participatory story work. NarraFirma would likely not have been possible without Cynthia first spending several years to create the latest edition of her textbook, since NarraFirma closely follows the methodology she outlines there.

How does NarraFirma fit into a bigger landscape of “competitors”?

NarraFirma covers a lot of ground in about forty virtual pages (as a “single-page” webapp) to lead people through the “Participative Narrative Inquiry” (PNI) process — which is not surprising given it takes a 700 page textbook to explain the PNI process. And NarraFirma still does not directly support everything involved in the PNI process. NarraFirma supports planning sensemaking workshops and recording the results, but it does not support what goes on in the workshops. NarraFirma could eventually having online versions of all the dozens of different sensemaking exercises or methods that are done in face-to-face workshops. The clustering diagram part of NarraFirma is just the beginning of such tools, as a proof-of-concept in that direction (and one that we’d be the first to say could be improved further).

However, as much as we’d say PNI is a great tool, PNI is just one approach to sensemaking out of dozens that Cynthia has learned about that. Looking at sensemaking broadly, these approaches range from the very old (e.g. the Chinese I Ching, the Native American Medicine Wheel, and the Italian Tarot, to name a few) to the modern (Structured Dialog Design, Appreciative Inquiry, Cognitive Edge’s Cynefin process, PNI, IBIS, Compendium, and many more). Cynthia has plans for a book and/or wiki covering all of them someday. Even Bucky Fuller’s “Comprehensive Anticipatory Design Science” concept is a form of sensemaking about “livingry” to some extent. The fields of legal arbitration, mediation, and even just conventional jury trials also include sensemaking processes of a sort. Likewise, legislative processes including public town hall meetings also include sensemaking processes (often ad-hoc ones).

We feel PNI works well for a range of situations, but there are plenty of cultural, practical, and domain-specific reasons to consider other alternatives for sensemaking. These different approaches to sensemaking could be seen as a bit like different architectural styles or different musical styles. People may prefer different styles of architecture or music for all sorts of reasons. As great as, say, much country-western music is, and even as no doubt some people listen to that almost exclusively, we would all be poorer if country-western was the only kind of music around. Likewise our built landscape would be a lot less visually interesting if everything was Greek Revival style. Likewise, a claw hammer can be a useful thing to have in a toolbox for a variety of task, but other types of hammers may be easier to sue and produce better results for other task — as are other tools of completely different types like screwdrivers, tape measures, and saws.

Why did NarraFirma take over a year to finish?

We originally thought the project would take a few months, after which I intended to apply for the Automattic JavaScript Engineer job (or similar positions). However, implementing the software took a lot longer than we expected. These are some of the reasons.

  • Some of the extra time came from rewriting the GUI a couple of times, first with the Dojo Toolkit and then with Mithril, to make it more maintainable (especially in the long term) by my wife, who has less programming skill than I do, but is still an experienced programmer on her own. Mithril is similar to React, but sleeker, and it does not have the Facebook patent license gotcha.
  • Some of the delay was from trying several different approaches to the data storage back end. (I’d still like another go at that someday to simplify it further, but as Jeff Atwood says, “tending your software garden” is never finished. At some point you need to ship what you produce).
  • Some time was spent converting much of the code base from JavaScript to TypeScript when the complexity of 100+ source files became hard to manage in pure JavaScript.
  • Some time was spent on reading (and experiments) to improve my understanding of other web technologies and their strengths and weaknesses in different situations. That’s how I ended up choosing Mithril over other possibilities such as Angular, Ember, Backbone, and even the excellent React.
  • Adding statistical results was more time consuming than I expected, as (surprisingly) there are not yet any complete statistical libraries for JavaScript (similar to Apache Commons Math or SciPy).
  • Finally, of course, the project requirements my wife outlined grew as time went on, and one NarraFirma feature led to another and another — sometimes requiring substantial architectural changes. For example, we decided that we needed to support (limited) multi-user editing using periodic polling for fine-grained changes.

In other words, NarraFirma was yet another example of how “Software is Hard“. :-)

“But the nature of software is that the problems are always different. You never have to solve the exact problem that someone’s solved before, because if software already existed that solved your need, you wouldn’t have to write it. Writing software is expensive. Copying software is cheap. Scott Rosenberg coins this as Rosenberg’s Law: Software is easy to make, except when you want it to do something new. The corollary is, The only software that’s worth making is software that does something new.”

Why did you keep at it?

Despite the goal posts moving ever forward, I kept at NarraFirma pretty much full time, learning a lot along the way, but burning up almost all the cash and credit (~100K) we had saved during the 2.5 years I spent contracting for NBCUniversal. It felt somehow appropriate to use that NBCU-derived money to help people share and make sense of their own stories via WordPress, instead of just passively viewing other people’s stories and “expert” interpretations of them on TV. And of course, we like to finish things we start, as a half-finished ship with big holes below the waterline is usually just a waste.

Mr. Furious: They must have ripped the “Q” section out of my dictionary, ’cause I don’t know the meaning of the word “quit”.

What are your plans now?

Now, thankfully, finally, a first version of NarraFirma is done-enough to release. Of course, we’re also now nearly broke :-) and not for the first time from working on our own open source software. :-) But that’s another story, and we’ve recovered from that before (see below).

The good news (for Automattic) is that I’m now tooled up and even more eager than before to work on more great (but-hopefully-paying-the-bills-this-time) JavaScript software for WordPress as well as other Automattic services for many years to come. :-) Especially considering likely college costs down the road for my kid. :-)

It would of course be great to win the Knight News Challenge and get some grant money to work on these sorts of projects like NarraFirma and Twirlip (Twirlip being an aspiration for a wide-ranging public intelligence and sensemaking decision support platform). But with 1000+ terrific entries in the current challenge, and our not having won the challenge before (despite one of my wife’s ideas getting to the semi-finalist level a few years ago), we don’t have much hope of funding from that source. We see our entries mostly as advertising and social networking. Update: We did not make the cut into the next stage with either Twirlip or NarraFirma. :-(

The NarraFirma software is now in a fairly stable state, and my wife should be able to maintain it on her own going forwards. We can even hope some of her clients might help pay to improve it a bit here and there. She did much the same when working on SenseMaker Explorer; the US government and Singapore funded the first versions (a big upfront investment of tax dollars), and then she continued to improve SenseMaker on each paying client project.

Did you have any financial justification for doing NarraFirma?

As a long-term financial investment, we hope that the NarraFirma software will increase demand for my wife’s consulting and training services (as well as printed book sales), but whether that investment will pay off in practice is still to be seen.

Why don’t you just get investors for NarraFirma?

We could try to get investors (beyond ourselves), but we don’t really see an exit strategy for them. Our revenues will be limited by the number of hours Cynthia can do consulting or training, which can’t scale. We could maybe franchise what Cynthia does, but that’s not something we have direct experience with, and she has also gone to great lengths like with her book and now the software to make such information freely available. We designed the software to be easy to host, so hosting is another problematical funding vehicle, and in any case, requires a certain scale to make sense.

The NarraFirma software is still more of a niche project than something likely to be as broadly adopted as something like WordPress. We can imagine tens of thousands of users someday, but that still is not much of a revenue base for a growth-oriented company, even if it might keep Cynthia very busy with consulting and training.

A couple of people who wanted to invest in the past in Cynthia’s work generally wanted it to be a proprietary product and did not get the open source approach (not saying we blame them, it’s a tough road sometimes as we know all too well). So, the open source emphasis would be another hurdle in finding investors. Also, even just one investor means having to keep the investor in the loop on decisions, perhaps with a veto on open sourcing things, so if the investment is not big enough, it is not clear that is worth it given other options. But a big investment brings us back to there not being a clear exit strategy…

Probably a better model for funding an open source NarraFirma than investors expecting a monetary return would be individuals or a foundation expecting a social return. We tried and failed with the Knight Foundation, but there are a lot of places out there. Cynthia has even helped some before with her PNI techniques. So, we may yet find such an “investor”.

But even if we do find such a benefactor, the other things I say later about the value of me working with Automattic on public intelligence and sensemaking decision support tools would still apply. Google has many millionaires working there because they feel they are making a difference. For the same reasons, working with Automattic is the kind of thing I would still like to do even if, say, Cynthia and I won MacArthur “genius” fellowships — fat chance, but you never know. :-)

The Shoveller: If we had a billionaire like Lance Hunt as our benefactor…
Mr. Furious: That’s because Lance Hunt IS Captain Amazing!
The Shoveller: Oh, here we go…
The Shoveller: Don’t start that AGAIN. Lance Hunt wears glasses, Captain Amazing DOESN’T wear glasses.
Mr. Furious: He takes them off when he transforms…
The Shoveller: That doesn’t make any sense, he wouldn’t be able to see!

How do you expect to make money from NarraFirma and “Working With Stories”? How does this relate to Automattic?

The short answer is that we expect NarraFirma and Cynthia’s book to increase demand for her consulting services, and for me to use NarraFirma in my portfolio to help get a job working for someone else. However, the long answer is a more complex examination of what it means to try to be an independent software developer trying to write free software.

Cynthia and I have written several pieces of software made available in different ways with different revenue results. In the later 1970s I was paid by the hour by a teacher with a software company to write some CAI software for the Commodore PET called PetTeach and related content for maybe a few hundred dollars total (thanks, Jack!). In the early 1980s I got some royalties (about US$10K total) from a video game I wrote on my own in assembly language for the Commodore VIC called “Intruder Scramble”; it was a dumbed-down copycat of an arcade game called Scramble a college roommate liked to play and who suggested I write something like that (thanks Mike!). At universities like SUNY Stony Brook and Princeton, I’ve taught and supervised students as an employee at university wages. Cynthia and I have released Delphi software that was about eight person-years of work total both as shareware (earning probably less than US$10K total between PlantStudio and StoryHarp, mostly PlantStudio) and under the GPL (for our Garden Simulator from the start, and later the other two products). To be fair to StoryHarp though, it was probably why I got my first IBM Research contract working in the Speech Group, since StoryHarp was a voice-activated choose-your-own adventure system. Also, PlantStudio could have been much more financially successful and had an enthusiastic following, but we were too exhausted working at IBM to keep improving it and its moment passed. Cynthia and I have done some fixed-cost projects for others that have ranged from making less than minimum wage in the end (or even a loss) to making more than $200 an hour (once, for a substantial Smalltalk project). I’ve done a lot of hourly consulting as a contractor (some with Cynthia) ranging typically from about US$70-US$120/hour (which used to be worth more in the 1990s) for various big companies like Nabisco, the Principal Financial Group, IBM, and NBCUniversal and for governmental organizations like DARPA and Singapore’s RAHS. Almost all that work was done via intermediaries who got their often huge cut, and sometimes ownership of the software results. I’ve sold Android software for $2 and $5 a copy in the Android market (making a few hundred dollars total after months of work). And I’ve written a lot of freely licensed work for the web (including a book length essay) that have earned nothing (although I hope have overall made the world a little better place). Cynthia has freely licensed her “Working with Stories” book. Cynthia’s book (which she has put about two to three person-years into total) has earned about US$3000 or so from mostly Amazon sales and a couple hundred dollars in donations from kind and generous people (thank you!) over the past two years or so, which is just about enough to cover the cost of what she paid the amazing person who did the index for the book. :-) With the exception of my early video game and PlantStudio, the only significant money we have made has always been hourly consulting, or, alternatively, well-planned well-defined well-estimated fixed-cost projects that are essentially the same as hourly consulting.

While things are changing, in the past, hourly consulting at good rates has been really hard to find on free and open source software. Still, we believe in free and open source software and content as the best way (ideally) to contribute to culture, so we keep trying to ride that unicorn. :-) Nina Paley’s video presentation for TED on “Why Copyright is Brain Damage” sums up a core truth about sharing information and culture (although Richard Stallman has long made similar but not identical points).

Copyright is based on the idea of “artificial scarcity” promoting the development of new works. While there certainly is some truth there to the model, the bigger picture is that copyright increases costs to a society greatly too, like the time cost of a permissions culture, the cost of lost works, the cost of worry by creators, the cost of censorship, the cost of police and jails, the cost of training people not to share from an early age. As Nina Paley’s animations illustrate in her talk, humans are creates that both receive and transmit information, and copyright generally shuts down that process. Essentially all new works build on what has gone before in all sorts of ways. And as with music (including via EvoJazz) it is relatively easy to systematically via computer create every possible variant of some small number of themes, which then is problematical if copyright is considered (in many cases) to be infringed if a work has a similar sequence of a few common notes. As Richard Stallman has said, when copying was expensive centuries ago, given the cost of printing presses and manual typesetting as a bargain for more writing, the public traded away a right it could not easily use for some benefit; but in the digital age of almost free copying, the public can easily exercise the right of copying, so the old bargain is no longer a good one. And the bargain has gotten even worse with copyright extensions, not better. As a rough analogy, and without intending to directly equate the vast human suffering under slavery with much lesser suffering under copyright, the laws surrounding slavery in the USA got worse and worse (for both slaves and abolitionists, especially with “The Fugitive Slave Act”) after Eli Whitney’s Cotton Gin technology made slavery more profitable, until an eventual Civil War. Likewise, the internet and digital media have made old works more valuable via a long tail of sales, which has driven a push to expand copyright, both to make money and to keep free alternatives unavailable which might distract users from paid media. This is done both by expanding law and by legal bullying. As just one tragic example, the inspirational movie “It’s a Wonderful Life” was forced back out of the public domain where PBS made it a classic and solely into proprietary hands.

Copyright is obviously a complex issue. Moral rights like requesting attribution and avoiding misrepresentation of an author’s intent are certainly more defensible. See legal scholar James Boyle’ “The Public Domain: Enclosing the Commons of the Mind” for a more nuanced view.

Between my wife’s free book (without which NarraFirma would not be possible) and NarraFirma itself, Cynthia and I have put about three to four person years of work into this effort. As an opportunity cost, that is about half a million dollars we’ve “invested” in free content and free software. We’ve enjoyed doing that work, and we think it is a good thing for the world. We do still hope that investment may pay off financially for us via increased consulting for Cynthia’s services (but even if it doesn’t, it was still a good thing to do for the world). That financial payoff could be from both projects being essentially increased advertising for her consulting services. It can also be by making it possible for Cynthia’s potential clients to do projects more inexpensively than otherwise because NarraFirma lowers the learning curve for Participatory Narrative Inquiry and speeds the actual process, and so increasing the potential pool of clients. However, those returns are very indirect compared to creating artificial scarcity for a book or software, convincing people they have to want it, and then saying “no, you can’t make a free copy” and threatening people with the police until the person hands over some money first before making a copy.

There are of course several ways to make money with free software beyond selling cross-selling consulting/training services or printed books — or begging mostly unsuccessfully for donations. You can upsell a free plugin to some otherwise hard to get version (as some WordPress plugins do, but we chose not to). You can sell backend services for plugins (like MailChimp) as a form of “hosting”. You can host the entire setup, like is done by WordPress.com and other WordPress hosting providers. You can sell related advertising in various ways (which usually then entail tracking users and their interests). You can sell services related to customization of the software itself or bug fixes. There are no doubt more angles on this as well.

All of these approaches have potential negatives as they generally create incentives to make software that, depending on the approach, is some combination of hard to setup, hard to maintain, buggy, poorly documented, hard to learn, hard to use, incomplete for typical tasks, dependent on a central proprietary service, and/or privacy violating.

Some of these approaches, like hosting (as with WordPress.com), are less problematical than the others. But as with WordPress.com and a few other big hosts, there is often a certain entry barrier and monopoly of scale that makes that profitable for only a few who are either early adopters (like Automattic) or late adopters with a lot of capital for advertising.

The WordPress ecosystem can provide a living for many people. One estimate I saw recently was a million people providing WordPress-related services either now or soon, which is an amazing good thing. However, I’d suggest, relative to the scale of WordPress covering 25% of the web, very few of them are full-time software developers (like I was for about a year writing NarraFirma). Even, say, 100 independent software developers making a living writing WordPress plugins full-time divided by millions of sites is essentially zero independent developers supported by the WordPress ecosystem (and I’d guess 100 full-time independent developers might be close to the number if that, if you discount full-time developers who otherwise work for places like MailChimp, Automattic, or wherever else that captures value via server hosting or which otherwise supports developers for a specific institutional mission). As time goes by, relatively few such people will even be system administrators making small plugins or patches here and there given the “cloud” and autoupdates. Most of the people making a living with WordPress will be probably educators, hand-holders, setup people, and people providing other related services (like custom editing or writing).

That is not necessarily a bad thing, since it is in the nature of a post-scarcity society that a tiny number of infrastructure-minded people can support the many in some area. The problem is that our society does not realize its economics should shift from “artificial scarcity” to “post-scarcity” in more and more areas.

Every once in a while perhaps a group might get some money to write software from grants. In fact, as is said in this article from 2008 on “Is Open Source the Answer To Giving?”:

“Mark Surman, Shuttleworth Foundation fellow, writes that open source is the answer to philanthropy’s $55 trillion question: how to spend the money expected to flow into foundations over the next 25 years. While others have lashed out at ‘Philanthro-Capitalism’ — claiming that the charitable giving of Gates and others simply extends power in the market to power over society — Surman believes that open source shows the way to the harmonious yin-yang of business and not-for-profit. Sun, Microsoft, Cisco, IBM, Yahoo, and Facebook are big backers of Creative Commons; Mozilla has spawned two for-profits. Open source shows that philanthropy and business can cohabit and mutually thrive. Indeed, philanthropy might learn from open source to find new ways to organize itself for spending that $55 trillion.”

I said something related in 2001. If such people getting grants are wise these days, they could write such software as WordPress plugins (when sensible). That is as opposed to, say, what the Omeka project did, which, while very admirable for its aspirations for user functionality, essentially tried to recreate WordPress instead of just making an outstanding WordPress plugin with the same museum collection management ideas. The same is true for Cynthia’s Rakontu project, which would have been better performing, easier to write, and probably much more successful overall as a WordPress plugin.

However, people who can go out and get grants (or the rare good free and open source software jobs at existing foundations) are again not usually the kind of “independent software developers” I am talking about. Don’t get me wrong — I’d be happy enough to do either of those (get grants or work at a foundation on free software done in-house), and I have tried to apply for both situations. There is nothing wrong with that. In the past, those kind of situations have just been hard to find though, given a focus by even foundations on subsidizing more proprietary content on the foolish notion the revenue from that will then fund yet more proprietary content, which it rarely does. Back around 2000 I talked to some foundation staff at the Beldon Fund (now defunct) about Linux and free software in relation to an IT job opening there when they has already committed strongly to Microsoft Windows and various proprietary software offerings. They apparently could not see any conflict between that choice and missions to help “achieve and sustain a healthy planet”. I’m not saying they made the wrong choice for what they focused on if the issue was just priorities and limited resources — just that it was really hard to get people back then to even see the issue was something to consider (and even now it can still be challenging). That may be changing as our culture shifts, as is suggested above, so my experiences are from many years ago, and so the number of free software jobs in foundation is no doubt many times greater than before. Likewise, more and more foundations are realizing that they get more for their dollars when their grantees release the results under free licenses. So, there may well be many more good opportunities to develop FOSS at foundations or non-profits they fund.

We’ve also realized that most “plum” jobs in the non-profit sector most often (though not always) go to well-connected people and so are hard to get. These connected people are often not your typical nose-to-the-writing-and-programming-grindstone more-introverted-side-of-things types like Cynthia or me. Such non-profit superstars (especially organization leaders) are also often from upper class families, with such people going from success to success over years (or sometimes even failure to failure), often with parents (or professional spouses) filling in any employment gaps with new cars and condos and trust funds and so on. Even Bucky Fuller was from a wealthy family and got a “small inheritance” at a critical time in his life. We’ve learned to read between the lines of profiles on various successful artists, writers, and entrepreneurs to look for the family money (it’s not always there as with Steve Jobs, but it often is, like with Bill Gates). And while Cynthia and I have both received some help from our parents over the years in various ways (thanks!), and it was a lot for them to offer in their middle class circumstances (double thanks!), it does not amount to what we have seen many others get whose parents (or spouses) were just in another social class altogether.

Despite what I said above about class and free software funding, it is true that Cynthia’s work as a professional has enabled me to work on free software and free content, and my professional work has enabled her to do the same. That has been a workable (if stressful) compromise throughout our marriage, although it probably would not be feasible for most free software developers. And there are no doubt many introverted hard-working programmers from middle class backgrounds who have made it into foundation-funded software development work — just not us. And if we had never racked up debt while trying to do our own independent free software development work, and also had been willing to put our kid into public school instead of homeschool/unschool, and also had been willing to relocate to near a big city like Boston or Washington D.C., then Cynthia and I could likely have had good jobs both working full-time with non-profit groups like Concord Consortium doing free software and free content funded by foundation grants and government grants (although probably on projects geared for schools and not informal education, and still requiring some good luck to get). So, I won’t say we did not have other options in retrospect and that we might not have done things better somehow — even if there may be chains of consequences, like if we had lived around Boston or Washington we would likely forever be on a treadmill of working for others to pay an expensive mortgage (or high rent) and never have an option like homeschooling or doing independent projects. But I will say from first-hand experience and extensive thought and reading that the currently available options for independent free software developers in the USA are often highly constrained — especially given how much money the USA pours into military spending, mainstream medicine, and formal schooling compared to independent alternatives for exploring security, health, and education (or other areas of life).

When there are lots of jobs in some non-profit area, like say, reading tutor jobs, this networking effect does not matter much as many warm bodies are needed. When there are a few jobs in an area that are highly desired (like free software jobs a decade or two ago), then these networking factors are likely to be more significant. This does not mean well-connected people are bad; just that being good at extensive networking requires a certain skill set, personal history, and personal proclivities, which are often (but not always) at odds with being a skilled programmer. Both sets of skills can be useful depending on the circumstance. Likewise, I don’t mean to imply Cynthia or I can’t get along with others or work closely in teams, or that we can’t market ourselves and our projects to some extent; my point is that there is usually a big difference between someone who would want to spend a year developing a software program like NarraFirma and someone who would want to spend a year developing a funding program for something like NarraFirma. :-)

No doubt there are a dozen people working at foundations right now who would love either to hire Cynthia and/or me or to throw grant money at us to write more open source public intelligence and sensemaking tools — people who may be desperately and fruitlessly searching for exactly us. :-) But considering there are 140,000 grantmakers listed in the Foundation Center’s directory, and the non-profit Foundation Center also uses “artificial scarcity” as a business model (starting at around US$400 a year), and every grant pursued takes time away from actually writing software (as has writing this essay), statistically, we will never find those people and they will never find us. :-( Still, if all charitable dollars went to open source projects, and a lot more people understood the need for tools for democratizing intelligence and sensemaking, then the odds would be greatly increased of a connection. Also, some of the Foundation Center’s work is freely accessible, so looking at “How Foundations Are Supporting U.S. Democracy” in more detail might be helpful for me to do (and I just discovered that link while writing this paragraph).

Grants do have some problematical aspects for independents according to Free Software Magazine suggesting why Cynthia and I probably won’t get grants even when we find like-minded funders (including from not having PhDs):

Grants provide money in advance, when it is most needed, both to provide for the material needs of the project and to pay researchers for their time, making them the most obvious way to fund any public good. We’ve funded university and institutional research this way for ages. Many precursors of free software products got started like this, including the code that became the Unix family of operating systems. Clearly this system can and does work.

However, grant money, once given, is extremely hard to get back. So when a person applies for a grant they must endure a gauntlet of tests, intended to prove to the granting agency that that person is willing and able to fulfill the promise they make in their proposal. If a grant is received, will the project be completed? How much money will it cost? Real research is full of unexpected set-backs and cost-overruns. Real researchers are full of optimism and unrealistic deadlines. The skills for research, development, logistics, and management are rarely found all in one person—good scientists rarely make good accountants, let alone good receptionists.

This encourages the granting agency to be very selective and take few risks with whom they fund. Researchers must have a proven professional background, track-record of honesty, and a reputation to protect. This is why funding by grants requires the use of large government, foundation, and university bureaucracies, and only the professionals who have climbed the career ladder to positions in these organizations have a serious chance of benefitting from them. The “solitary inventor” is indeed dealt out of this game, just as Eisenhower predicted.

Now, at this point, you might think I’m going to call for Automattic to share its revenues somehow with independent WordPress plugin and theme developers, or something like that. It’s an interesting idea, but in practice, I don’t think it would work. For example, how would you attribute revenue between something like NarraFirma that ultimately took several person years to produce and maybe just a small number of communities might use (but get a lot out of) and some plugin that took a couple weeks to write (like perhaps the outline plugin I used here) but is used on thousands of sites on many pages? I don’t know the answer to that, and I doubt anyone else does either. Dividing a “cake” can be very hard to do, given we may all have different notions of “fairness” or “accountability”. See the book “Policy Paradox: The Art of Political Decision Making” by Deborah Stone for a detailed discussion of this “fairness” issue.

Still, I am nonetheless going to call for Automattic to do something, and that is to try to understand post-scarcity economics better, and to, if appropriate, call for systemic political change like a “basic income” as one way to address this issue among others. Other possible ways are greatly increasing the gift economy, helping people towards better subsistence (like learning through reading and writing WordPress sites), and supporting better governmental planning via democratic participation (including via public intelligence and sensemaking tools as I propose here). I talk about those sorts of ideas in a few places like this essay on “Beyond a Jobless Recovery” and this video “Five Interwoven Economies: Subsistence, Gift, Exchange, Planned, and Theft“. Automattic could also call for all projects funded in whole or in part by charitable dollars or tax dollars to always be completely free and open source, like by building on this letter I wrote over a decade ago on that topic. I don’t know what political lobbying it would make sense for Automattic to do, but I’m asking for people at Automattic to at least have a coherent justification for not sharing its revenue with plugin developers and to do something else in accord with various ideals of fairness, given Matt Mullenweg has repeatedly said it is the WordPress plugin ecosystem is a big part of what makes WordPress so special.

In the USA, a basic income would in part compensate citizens (including Native Americans) for US government enclosing most of the land and so making subsistence production impossible for most people, as well as (like C.H. Douglas wrote about with Social Credit) the idea that our culture is a common inheritance and most productivity today comes not from labor but from culture and past experimentation. If 50% of the US GDP was distributed as a basic income (likely replacing social security, unemployment insurance, much public schooling, all of copyright law, and more), that would be about US$2000 per person per month. For our family of three, that would be about US$6000 a month, which is about in practice what we live on, given some extra expenses for buying mostly organic food (for health reasons and to support organic farmers), from homeschooling/unschooling, and various business expenses and charity contributions and such. Such a basic income would mean Cynthia and I (and maybe our kid someday) could write lots more free software and free content without worrying about figuring out how to “monetize” gifts to the world or figuring out how to write software full-time (which takes a lot of concentration) while *also* providing tangentially related services full-time.

A basic income might also help more people in the USA adopt a hunter/gatherer mentality by having more “free” time for politics, hobbies, and family. Consider “How Hunter-Gatherers Maintained Their Egalitarian Ways” by Peter Gray, where he suggests:

The hunter-gatherer way of life, unlike the agricultural way of life that followed it, apparently depended on intense cooperation and sharing, backed up by a strong egalitarian ethos; so, hunter-gatherers everywhere found ways to maintain a strong egalitarian ethos. Now, back to the main question of this post. How did hunter-gatherers maintain their egalitarian ways? Here are the three theories, which I think are complementary to one another and all correct.

Theory 1: Hunter-gatherers practiced a system of “reverse dominance” that prevented anyone from assuming power over others. …

Theory 2: Hunter-gathers maintained equality by nurturing the playful side of their human nature, and play promotes equality. …

Theory 3: Hunter-gatherers maintained their ethos of equality through their childrearing practices, which engendered feelings of trust and acceptance in each new generation. …

… As I have explained in my published writings, anthropologists commonly identify two varieties of hunter-gatherers. By far the largest variety, in terms of number of different cultures identified, are the “band hunter-gatherers,” also sometimes called “simple hunter-gatherers.” The other variety, sometimes called “collector societies,” or “complex hunter-gatherers,” includes the Kwakiutl of the American Northwest Coast and the Ainu of Japan. These are sedentary societies that live on some rich, localized resource, commonly fish. When anthropologists use the term “hunter-gatherers” or “foragers” unmodified, they are usually referring specifically to the band variety. The collector societies are more like agricultural societies in many ways. They have relatively dense populations, hierarchical social structures, chiefs; and are often warlike. Most archeologists believe that band hunter-gatherers long predated collector societies and are more likely to represent the normative way of living of our distant hunter-gatherer relatives. However, a reasonable case can be made that over a long period of history both types of cultures existed. If true, this may help explain why our genetic tendency toward dominance behavior, which we inherited from our ape ancestors, was not selected out over the course of our hunter-gatherer history.

The key differences between the two in terms of effect on cultural moral ethos is that the collector societies, like agricultural people, are sedentary, accumulate wealth, and have a resource to defend.

BTW, defending the WordPress.com money-producing “resource” over time may make Automattic become more like “complex hunter-gatherers” than “band hunter-gatherers”. One can already see that in Scott Berkun’s book, which documents Automattic’s move away from a flat model to more hierarchical teams. I don’t know what specific alternative organizational form Automattic could pursue, but I can hope that stigmergic and chaordic processes mentioned later could help with that somehow.

While I feel investigating and perhaps lobbying for  for post-scarcity economics would be a big win for everyone (whether Automattic did it or anyone else), it’s a fair criticism that such an effort might take decades — and in any case it would be tough sell to current Automattic investors enmeshed in the current financial system. So, here is a more down-to-Earth easily-doable suggestion for Automattic about helping direct some of that US$55 trillion mentioned above to good homes. :-) Automattic could hire several experienced fund-raising development directors who would then (as a team) provide a free service to independent WordPress plugin developers to help connect them to foundation grants and government grants somehow (doing most of the leg work, including directly contacting foundations on their behalf to pre-qualify ideas and save software developer time). This team could also otherwise help developers brainstorm how they might fit their software development efforts into existing foundation, governmental, or non-profit missions somehow or connect with fiscal sponsors (for grants that required 501-c-3 status). It might perhaps make sense to also have another team to help with WordPress plugin crowdfunding efforts.

Basically, this “financial development” group would do all the things Cynthia and I perhaps should have done when we started our Garden Simulator, to follow up some basic suggestions about from well-wishers. For example,  an actual suggestion by someone with years of fundraising experience for funding the garden simulator, that I probably foolishly ignored, was to approach a group that cared a lot about tomatoes and to make a first version of the software to help people learn about growing tomatoes using funding from them. Still, given the low probability of success and our low skills at fundraising, that educational software might never have gotten written if we had focused on fund-raising first to the exclusion of just coding away — but we will never know.

A team like this at Automattic lowers the risk for WordPress plugin software developers to at least explore grant funding opportunities. Such a group could also identify undeserved niches in the WordPress ecosystem with likely foundation or government grant money (or possible crowdfunding) available to fill the niche via a WordPress plugin, and point developers to such opportunities via some publicly maintained list. Since this service might be in huge demand, perhaps the service could just be very cheap instead of free, like US$100 a year for active support or perhaps developers would have to already have at least one plugin hosted at WordPress.org. Or there might be some other reasonably low bar for developers to show a minimal level of sincerity and commitment before Automattic development directors start banging on foundation doors on their behalf. There might also be some limit to the size of the development group getting substantial direct advice, since otherwise a few big companies with their own in-house fundraising development staff might monopolize the service. I’m not sure on that last, given the potential for an in-house Automattic team to learn from any sized organization; there might be some other way to address that concern if it appears in practice.

I don’t know all the details of how this would work or all the possible consequences or other concerns (like privacy or competition or whatever). If such a service was too broad, like supporting all WordPress content developers (who could write content on any topic), it might also just devolve into being another Foundation Center providing lists of places to contact based on some general search (although, circumventing the Foundation Center paywall for the WordPress community might still be worthwhile just by itself).

Alternatively, or in addition to the above, since a lot of non-profits and foundations already use WordPress, there could perhaps be a new WordPress plugin and Automattic-hosted backend to just bypass the Foundation Center altogether. Groups could enter their interests as funders or developers into the plugin and the software helping propose matches and track progress towards actual funding. Ideally such a plugin would be available at WordPress.com. In that case, my local historical society could use it, potentially (subject to board approval) perhaps to offer the possibility to act as a fiscal sponsor for other developers doing historical-related software. Because there is an obvious conflict-of-interest if I as a volunteer trustee got paid by our non-profit to develop WordPress software, it might be great if there was a service where I could connect with another non-profit to provide independent fiscal oversight if I otherwise could get a grant to fund a year of my time to, say, turn Omeka into a WordPress plugin to support the collections in our own museums as well as those of other historical societies.

However this idea was implemented, as with Akismet, Automattic could be a hub of crucial information to the WordPress community, but in this case about what foundations to approach instead of what spammers to avoid. In this way, Automattic might leverage a few hundred thousand dollars a year spent on non-profit fundraising expertise (or in-house plugin development) into potentially hundreds of millions of dollars a year for WordPress plugin developers.

However without a 50%-GDP-egalitarian “basic income” as social security for all from birth to pay our bills for food, electricity, mortgage, car, and so on, and without grants, Cynthia and I still need to do something else in exchange for the goods and services we so thankfully have available to us to meet our needs and sometimes wants. Our strategy at this point is two-fold. We hope Cynthia might be able to bring in maybe $30K a year from part-time consulting at $100/hour (while still homeschooling our kid) given she is a domain expert in Participatory Narrative Inquiry. We hope that I could use NarraFirma to add to my portfolio to get a more typical job. Ideally that would be a “dream job” doing open source decision support tools and such working with others with similar interests, working from home so commuting time saved could help with homeschooling/unschooling. But statistically probably I will not find both — or maybe either — unless I land a job at Automattic as I propose here (or maybe one similar elsewhere like perhaps Craigslist.org or wherever else might potentially want sensemaking tools for the masses). If I can get a job to pay our bills, then in about three years, Cynthia maybe could earn enough from side consulting to pay off the recent credit card debt we ran up to finish NarraFirma, and we would have worked our way up to being only broke and without retirement savings. :-)

From a purely financial point of view, that is a pathetic return on about a half million dollars of financial opportunity costs incurred over the past few years for Working With Stories and NarraFirma. It is an even worse financial return considering previous independent software projects we have done. If we had just worked at companies making proprietary or monopoly products, and put that money into stocks or real estate or even the next “Angry Birds”, maybe we’d be “making money in our sleep” (which is supposedly the only way to grow “financially obese” to call it by James. P. Hogan’s term). Instead, at best, such an investment only sets us up to do more work mostly for others (and likely at their ultimate direction in my case). We generally like our programming, writing, and other consulting work (even with ups and downs and frustrations), but it certainly has been life on the edge for a very long time, which is wearing emotionally and physically.

Now, someone more socially savvy than us might well point out all the things we could have done better. And I would not disagree. Spending so much time around universities and graduate schools may well have been a huge mistake as well. We made a lot of dumb moves. But we were still some of the smartest programmers around (or maybe not :-) , and some of the most committed to FOSS over the long haul — even if there are others who are greater than us in both ways. And there were two of us with some complementary skills. If we as a couple have had so much trouble supporting ourselves doing independent free software, what about all the rest of the developers out there on their own who are not quite as smart or are not quite as committed or whose marriage are not quite as strong (or are just non-existent) but who would still like to be writing significant GPL’d WordPress plugins about issues they care deeply about?

All that being said, would I do it all again? Yes!!! :-) I’m thankful we managed to do it the first time. Although obviously I would try to do it all over again a bit wiser, including by trying to create strategic alliances sooner with groups with common interests. :-) And ideally with having more kids along the way to bring even more joy and purpose into our lives.

 


More about changing NarraFirma from Dojo and JavaScript to Mithril and TypeScript


tl; dr We changed underlying technologies to make NarraFirma easier to maintain


Did it take a long time to make a major GUI switch from Dojo to Mithril?

While deciding whether and how to change the GUI took a lot of thinking and learning, the actual process did not take that long. That was because I had set up an architecture where most GUI screens were defined by JSON data structures interpreted by a GUI panel builder assembling a set of pluggable components rather than being defined as code that constructed the GUI directly. That choice was driven in part by my wanting to support my wife outlining the design in an easy-to-read-and-write specification language rather than having her try to learn JavaScript at the start of the project.

Why did you design the system with a panel builder?

I had started using Dojo (including Dijit for widgets and GFX for drawing) several years before for a graphical app that ran on CouchDB. So it was the first thing I turned to. Dojo does have many great features including terrific data grids. Dojo pioneered much of what we now take for granted in web JavaScript development. Dojo 2.0’s move to TypeScript influenced my own decision to move to that later. But I also had some misgivings about Dojo from the start, both in itself and also because of the large numbers of JavaScript GUI libraries out there, the rapid change in that area, and how hard it was to pick one from all the alternatives given all the overlapping feature sets.

As a proverb says (Sun Tzu?), when confronted with 100 options, the best choice is always to run away. :-)

One way to “run away” in programming is to create an abstraction layer or layer of indirection. I did that by using a specification-based approach to define much of the GUI. A tradeoff in that specification-based approach to hedge our Dojo bet was more machinery to support the abstraction. Excessive abstraction can itself become a problem in making designs harder to debug, even as abstraction might make some other things easier to understand or specify. And in practice, that extra layer of abstraction probably caused its own delays as well, both as I re-engineered it a couple of times to make the abstraction clearer and also from debugging it. It also constrained the look of the GUI in some ways.

What issues did you encounter with Dojo?

First, I don’t want this to be seen as Dojo bashing. Dojo is a great toolkit with a lot of good components. Dojo has done a lot for the web. I am thankful for all the hard work the Dojo community has put into creating and maintaining it. Dojo can be a good choice in some situations, especially for people already familiar with it who have been using it for a long time. But I still found Dojo-based code fairly difficult to work with for various reasons. Not all those reasons were Dojo-specific either, as some relate to the general issue of DOM manipulation on the fly which one can find in JavaScript projects base on Backbone, Angular, Ember, Enyo, JQuery, and so on (compared to vdom approaches like used by React, Mercury, and Mithril).

It became clearer as I continued to use Dojo and read mailing list posts and so on that as the web has matured over the past decade in terms of web browser getting better, especially with older versions of IE dying off. So, many of the features of the Dojo toolkit intended to isolate developers from compatibility issues are less needed given common browser support for various standards. With those changes in web browsers, it was not clear that the cost of using Dojo as an extra layer between me and the browser was worth it any longer.

For example, D3 uses the now common SVG standard, which is one reason I switched to it instead of continuing to use Dojo’s GFX. GFX is a lowest-common-denominator abstraction over several web drawing technologies including SVG, Canvas, Silverlight, and VML — but that cross-platform support just is not so important if you focus on supporting mostly just modern browsers. There was also a learning curve issue; there are tons of tutorials on using SVG and Canvas, but only a few on using GFX (and those may be out of date given changes in Dojo and GFX).

There are also now many mix-and-match options to various web challenges. Dojo offers relatively easy-to-use GFX-based charting which I used at first, but it was not as flexible as D3 (or several other charting options out there I evaluated but I did not use). Dojo has good solutions for many issues, but they are not always the best. Billing itself as a “toolkit” not a “library” or “framework”, Dojo was itself mix-and-match, but in practice there was still a significant needed core almost everything assumed. And even using batteries-included Dojo, I found I had to turn to underscore.js (and then lodash) for things like “startsWith” for string comparison — for some reason “startsWith” was a feature Dojo had removed at some point to save supposedly space (despite endless other features).

To be fair, though, I was learning a lot about how to shoot yourself in the foot with JavaScript throughout this entire process. :-) So, it’s hard to pin all the difficulties I wrestled with on Dojo, Dijit, GFX, and so on. I suspect Dojo gets a lot of ill-will from people who pick it but are still just going up the JavaScript learning curve; however, because Dojo adds an extra layer of complexity, it does sometimes feel like it makes just about every JavaScript error you can make harder to debug. And of course, Dojo has its own (very small) share of maybe bugs and/or gotchas (but to make up for the issues any library has, it has a great community). On the plus side of all that, I got much better at debugging code using complex JavaScript libraries. :-)

A major reason I stuck with Dojo so long was its accessibility and internationalization support. But ultimately I decide that something like Mithril would likely be a11y enough because it was close to plain HTML, and browsers were good at that (even if it could be better). I also ended up using some more custom internationalization support instead of Dojo’s built-in support, which requires indenting translatable tables in two different ways between English and other languages. NarraFirma’s i18n support is still not quite done, but it is something my wife has done on her own before, like for Rakontu.

How did you go about switching from Dojo to Mithril?

My growing feeling was that if I found Dojo hard to work with, especially all the complex GUI-to-Model dependencies, my wife would find Dojo-based code very hard to maintain. I looked at a lot of different options, and ultimately decided on switching to Mithril. Granted, I knew a lot more about the HTML DOM at that point, so I did not feel so much like I needed the “training wheels” Dojo and Dijit provided as it tried (and not always succeeded) to hide a lot of the DOM behind the Dijit library. It also was so much easier to find documentation on how to use the DOM directly compared to understanding how to interact with Dijit widgets to get them to use the DOM for me.

So, even after the Dojo GUI was more-or-less working well (and maybe 80% done), at some point in June, after a lot of research and experiment, and after previously converting the project to TypeScript to make refactoring easier, I made the switchover from Dojo/Dijit to Mithril for the main application’s GUI.

That full changeover was done after first making a lower-risk Mithril version of just the survey-taking code as toe-dipping — and the water felt fine.

As a partial move away from Dojo months earlier, I had also switched from Dojo’s GFX to D3 for graphing, which had worked out OK — even as I had been reluctant to lose some of GFX’s broad cross-browser support. But since almost all modern browsers now supported SVG (or can be polyfilled), using D3 with its SVG requirement instead of GFX seemed acceptable in a way it might not have been years ago with earlier browser versions. Also, by taking the lowest-common-denominator approach, GFX had its own limits (including some difficulties with touch events).

Switching to D3 and TypeScript were fairly low risk bets, as was doing survey collection in Mithril. By contrast, the full switchover for the entire main application’s GUI from Dojo to Mithril still was a big risky change taking some time to do with unclear benefits other than maintainability/learnability in the long-term given things were working under Dojo/Dijit/GFX/dgrid. The codebase’s internal level of complexity, number of layers, difficulty in debugging, and internal learning curve just felt too high to me (even having already gone up that learning curve myself).

We had by then begun letting our credit card balances run up to preserve cash in the bank, so it was very questionable to do any major changes that would take time and so make finishing the first version of NarraFirma take longer and delay me eventually getting a “real” job. And with our past IBM connections, and with IBM having invested in Dojo, moving away from Dojo seemed like it might preclude using NarraFirma as a portfolio piece to get a job maintaining legacy Dojo code — when I was sure there were likely many such jobs that might pay well (not that I especially wanted them, given the frustration level, but such jobs were better than losing our home). And of course, using Dojo had been my original decision. So there were plenty of reasons not to switch. :-)

But, I also knew that once I took on other work, Cynthia would have to maintain the NarraFirma codebase. Experienced programmer as she is, I just could not bear to leave her with a mess of GUI-to-model interdependencies or the frustrations of learning Dojo’s complex ecosystem (as nice as parts of Dojo can be when you know them, and as much as the Dojo community has moved web development forward).

It ultimately was a tough call made out of concern for issues like love, beauty, simplicity, elegance, and joy — a decision and subsequent implementation done while watching our bank balance spiral downward and our credit card debt spiral upward.

The Bowler: See now, this is why mad scientists are generally less desirable than your common or garden variety scientist.

What has been your experience using Mithril compared to Dojo?

Mithril in general has indeed been a joy to work with, as it runs very close to the HTML DOM. That makes it pleasantly easier to learn and use than libraries like Dojo, which attempt to hide the DOM but end up leaking it out anyway in practice. In those leaky cases, you still have to learn about and sort through DOM issues, as well as wrestle with the leaky and maybe buggy and incomplete abstraction. You still have to learn about CSS to style components and adjust their behavior. Once you know the DOM and CSS well, extra layers of abstraction begin to feel like clutter, unless they have some very clear benefit.

Like React, Mithril uses a vdom that closely parallels the real DOM. As with a typical 3D video game redraw loop, it is generally easy to reason about redrawing the entire page with Mithril when anything changes in the data model — relying on the vdom to minimize actual expensive changes to the actual DOM nodes.

The harder-to-maintain alternative that is typically used with Dojo and most JavaScript frameworks (e.g. Backbone) is trying to change only exactly what is needed in the DOM when the data model changes (typically using jQuery to make the actual change). These changes are triggered as needed by setting up potentially complex and often brittle dependencies between data model objects and GUI objects (as typically with Dojo widgets). It can usually be made to work OK. However, writing such code well gets hard when, say, two widgets can affect each other as well as the underlying model and you need to suppress “ringing” in the interface. Situations like that (and worse) can be hard to debug when things go wrong.

I say that as someone who has been setting up dependencies between data models and GUI widgets on-and-off for about twenty years since first doing that in Smalltalk, where such complex dependencies probably originated. :-) React’s Flux architecture (using single directional data flow) is intended to address that sort of issue. NarraFirma uses something similar, where changes to the triplestore trigger redraws in Mithril, as do most network messages. In this case, the Mithril/React/Mercury+vdom approaches with a Flux-ish architecture seem a step up from Smalltalk’s classical way of doing things.

Had I started with Mithril, knowing what I know now, I might have coded several things more directly in it. “Running away” from a decision has its price, even in programming. Ultimately, even with extra layers of abstraction, you still have to write code that actually does something; extra layers of indirection can make that fundamental code harder to understand, harder to debug, and harder to expand in new directions.

Why did you move the codebase from JavaScript to TypeScript?

I picked TypeScript over several other possible JavaScript alternatives I looked at because TypeScript kept as closely to JavaScript as possible (so, fewer extra layers to go wrong), while TypeScript was still significantly easier for writing and maintaining complex systems. I also knew I could easily ditch TypeScript and go back to plain readable JavaScript if things did not work out compared to some other transpilable-to-JavaScript languages.

TypeScript has its own issues, but overall it has been a good improvement over plain JavaScript. It rarely feels like it is in the way or keeping me from doing anything I might want to do. The biggest challenge is integrating third-party libraries, but between Definitely Typed and this example trick for writing a stub typing, I’ve found the integration not too much work for the benefits it brings.

Still, I’m not against using other languages that compile to JavaScript as the assembly language of the web. Many, many people use languages like C/C++, D, or Delphi that usually compile to regular assembly language to great success, or other languages like C#, Java, or Smalltalk that usually compile to byte code. So, down the road, especially given improved source maps, asm.js and native client, and ever faster computers, I expect we’ll see lots more people compiling to JavaScript.

Moving to TypeScript was a low-risk high-reward decision for me to make at that point in the project, given increasing complexity. I also made that choice to make it easier for my wife, who did not know much JavaScript, to work with the code base. That is because TypeScript IDE tooling, like the Palantir plugin for Eclipse, can make even plain JavaScript easier to learn and use, given auto-completion suggestions. Also, I wanted to get really good at working in JavaScript even if I compiled to it later, and using TypeScript helped me continue down that path (but with less pain — given a project whose own source code plus library source code was weighing in at around two megabytes of text).

While there was a small hurdle to start compiling the code under TypeScript, I did not have to put typings in place for everything at once. There are some parts of NarraFirma for which it might be good to add typings for down the road or to convert to idiomatic TypeScript class definitions to get better IDE code completion and search support. I generally just add typings or rewrite classes incrementally as I go along, mainly when I start debugging or improving some part of the code.

Are there any things you lost in the transition or benefits that were less than expected from the conversion?

Of course, even the best things in a programmer’s life, like Mithril, can have their design challenges and surprises which consumed many hours to understand, document, and work around

And there were some Dojo/Dijit/dgrid features we sacrificed. For example, our Mithril grids are still not quite as nice as the Dojo dgrid grids, since the table headers don’t stay in place when you scroll the table. But we decided we could live with that for now (and maybe fix it later) for the benefit of being able to move away from Dojo and Dijit and leave behind model-to-view dependency management completely.

Still, it is not quite 100% true that we left explicit dependency management behind. To make the Mithril grids perform better, I felt I still had to do some data caching. So there is still a bit of extra one-way model-to-view-dependency complexity when underlying grid data changes (requiring a call to “GridWithItemPanel.prototype.updateData“). But that is not much dependency management compared to the previous approach, and it’s isolated to just a few easy-to-find places in the code.


More about how NarraFirma works


tl; dr NarraFirma uses TypeScript, Mithril and D3 for the front end, AJAX with JSON for the back end; works with WordPress or Node.js; data (including changes) stored as triples; 15-second polling; Automattic audition PingPong server suggestion


What is a brief technical overview of the NarraFirma application?

The NarraFirma singe-page web application has more than forty virtual “pages” defined in TypeScript. It uses Mithril and D3 for the front-end GUI, which communicates to a back end using AJAX with JSON. Pages can include editable tables and editable 2D clustering diagrams. The application back end can be either WordPress (via a PHP plugin) or a small Node.js server application in JavaScript (this was especially useful for development testing). The back-end code is maybe 1% of the front-end code in size. The administration GUI for the WordPress plugin is also implemented using Mithril.

How is data stored?

Data is primarily stored as messages consisting of triples stored in the WordPress database (one table per project, with one message per row, with database indexes). Under Node.js, data is stored in flat files (one directory per project, one file per message, with indexing and caching in memory).

What is the advantage of a message-oriented approach?

This message-oriented approach supports several users collaborating at once to modify the same “page” and quickly see updates without a page reload. (It is subject to polling delays, however, and has other limitations. It would be great to improve on these in future WordPress versions).

I see some parallels in how this works to Automattic’s Cloudup approach of sharing streams, if you think about how each message (containing a triple to add to a triplestore) could be part of a stream of items. It is likely the NarraFirma code could be adapted to work directly with Cloudup, as a demonstration of new ideas for using that service. Automattic’s Simplenote and Simperium products also overlap to some extent with these sorts of decoupled ideas.

I would like to build a variety of open source JavaScript applications to work with those various Automattic back ends, which could increase demand for Automattic products. However, I’d still lobby for having open-source versions of the back ends available as well for local hosting — in the same way as WordPress is conveniently and securely hosted by Automattic and also hostable elsewhere (even locally on a laptop).

I suggested some ideas for such applications (for Civic Sensemaking) in a Knight News Challenge entry called “The Twirlip civic sensemaking project.” I would put my heart and soul into making those things work well, if they were open source.

How does the user tell the application to save data or load data?

The NarraFirma application has no “Save” button. Changes are automatically saved to the server by sending a message about the change, usually when the user leaves a modified text field or releases the mouse on a drag of a clustering-diagram item. This was a design decision midway through the project (which took some reengineering). Earlier versions had a save button on each page and warnings about unsaved changes.

Likewise, NarraFirma has no “refresh” button. Changes other people make to the same project are automatically displayed on the current page after a polling cycle completes and the client receives any outstanding messages from other users (including new survey results) related to the current project.

Undo is not yet implemented, but that should be not too hard to do, because the application has available locally every change ever made to the project (by anyone) in time order.

How often does the GUI poll the back end for changes?

Polling for changes is done about every fifteen seconds (for now) to reduce server load. Polling is not done using the standard WordPress heartbeat, but that could be possible. Polling also shuts off temporarily to reduce server load if the page is not currently active (the visible tab) in the browser.

In theory, socket.io (a wrapper with fallbacks for using WebSockets) could remove the need for polling entirely by using an additional server process, producing low-latency near-real-time updates to pages. Collisions between different users’ edits are not currently reported by the software, but that would not be too hard to do, and I have some prototype code that does that.

Multi-user chat about the current project is another feature we would like to add. If one accepts up to a fifteen-second delay in seeing another person’s message, basic chat would be easy to add (except maybe for the issue of where to put it in the GUI). But in practice, fifteen-seconds delay is probably too long, and also would prevent adding features like “the user is typing” indicators.

We’ve also run the system with three-second polling for fast response in multi-user situations, but the load is of course greater on the server. Ideally, if not using socket.io, we would like to make the polling cycle time adaptive, so that when two users are on the same page or are engaged in active chat about a project, polling cycles would shrink, making collaboration smoother. If people stop making changes to the same thing, polling time could expand again.

Bowler: (to Spleen) What is this thing you were handling before?
Spleen: That’s a Shrinker.
Bowler: How do you know? Oh a manual, cool.
Heller: That’s a high-temperature fabric adhesive liquid projector, based on simple dry cleaning technology. You aim that at a guy, and, I’ll tell you something: His clothes get so tight he can’t even breathe. I can trick that out with a clamshell holster. Comes with a leather carrying case, it’s got a wad cutter and a full warranty.
Bowler: Nicely done.

Are there any tradeoffs with the message-oriented approach?

Keeping a copy of all messages in a project locally within the browser limits projects to a size a web browser page can hold in memory. The need to retrieve all the remote messages for local searching also limits project size to what users will tolerate while downloading all messages at startup.

The benefit of this approach, however, is very fast response time to user requests once the application loads, since all project data is cached locally. There are some other nuances to the storage system to improve startup performance and reduce memory use related to “topics” and supporting requests for the latest item in a topic (used mainly by the survey-taking client application).

This particular approach for collaboration may not “scale” to “big data”, but it can work well for up to several hundred stories and a thousand or so edits per project, which is in the range of what most NarraFirma projects would be. As Malcolm Gladwell has said about the limits of big data: “More data increases our confidence, not our accuracy.” NarraFirma is more about accuracy than confidence, so it is complementary to big data — although if you can only have one, I’d suggest NarraFirma. :-).

The biggest practical limit currently with NarraFirma as far as project size is that it takes about a second to download a bundle of 100 messages/edits/stories from a shared WordPress hosting from a remote host over the internet. Each of those incremental changes is stored in a row in the database, and must be retrieved, modified (to add sender information), and packaged together. Faster servers and networks might increase that speed by a factor of 10 or more, but there is still going to be a limit related to that and the patience of someone starting up the application. The current example project with 80 stories and about 1000 edits defining a filled-in NarraFirma project takes about 15 seconds to start up from shared hosting (although you could then leave NarraFirma open for days and it would track any changes without ever needing to refresh the page).

Are there any future changes you might want to make to the message storage approach?

The network/server delay is a bigger limit in practice to usability than memory or CPU limits in a typical laptop browser. I have some further ideas to optimize the network/server performance by simplifying the way data is received and transmitted by removing some additional data added (a message flow trace) which requires sorting JSON object fields recursively to create canonical JSON (used to create a consistent object ID), but I have not yet tried them. In any case, eventually, larger collaborative tasks would require searches passed to a backend triplestore, relational database, or other data storage back end, or alternatively, would require storing retrieved data in a local IndexedDB database where any loading delay delay would only ever happen once.

Are there any potential gotchas in the NarraFirma approach to data exchange?

The NarraFirma system as-is relies on the client’s clock to be reasonably in sync (within ten seconds or so), because every change (typically adding a triple to a triplestore) is labelled with an effective time, and the latest change always wins to define the project state. The server will reject changes labelled as being significantly in the future, but code is not yet active to report on when client clocks are significantly behind (although some commented code for that is in the code base).

Earlier versions of NarraFirma/Pointrel would use the web server to timestamp changes, on the assumption servers generally have more reliable clocks. The NarraFirma/Pointrel code may have to go back to that approach eventually (as least as an option), or do something else like rely on the server to supply timestamps if a significant clock discrepancy is detected.

There are various possible features as above and others that we would like to implement or improve, but the system is usable as is, especially in the context of a small group working together on a project supported by ongoing chat via IRC, Skype, Slack, or other means.

How is NarraFirma built?

The NarraFirma WordPress plugin releases are built using Node.js, by running “npm run build-wp” from the command line. That runs some script commands defined in package.json which use RequireJS’ optimizer to do the packaging. That approach to using npm as a build tool (as opposed to Grunt or Gulp and such) was inspired by reading this blog post. I had considered Browserify and Webpack for packaging, but decided to stick with RequireJS for now (and Dojo’s loader before that) to make developer testing easier without a build step (other than a build process that goes on behind the scenes from using Eclipse’s Palantir TypeScript plugin, which generates updated JavaScript output after edits to TypeScript source files).

How might local storage ideas apply to something like WordPress itself?

The scaling issue mentioned earlier is a bit like how we use all of bicycles, motorcycles, cars, buses, trucks, trains, airplanes, and ships to transport people and cargo, depending on the situation. For many small group collaborative tasks, if you divide up the data requirements appropriately, a thousand or so changes is all you need to support them (like editing a small spreadsheet, a white board, an outline, or a shared document). It would not surprise me even if the majority of WordPress blogs out there have less than 100 pages/posts and less than 100 edits per page/post and (except for media) might be loadable entirely within a browser’s memory in under a minute over a typical network. Maybe that is not quite practical right now (what if you close the page accidentally and need to wait a minute again to download everything except the pictures?), but it might be soon.

If IndexedDB were to be used with such an approach, I would venture a guess that 80% of WordPress websites could relatively easily have a working copy kept in an admin’s web browser (something perhaps not possible when WordPress was created). Incremental changes could be pushed up to a server, similar to how NarraFirma works. That server would then render the changes as HTML to non-admin viewers and could also distribute the incremental changes to other admins when they connected. Such changes could also potentially include JavaScript-powered plugin apps.

That approach might make editing a WordPress blog very snappy, which might help WordPress become the premier personal information management tool for everyone, eventually increasing demand for Automattic WordPress hosting. Why stop at 100% of the web, when WordPress could conquer a big part of the desktop too? :-)

A new open source project by someone else called “CloudWall” based on PouchDB and CouchDB does something like that as far as storing data and apps locally in a browser. However, there is no reason a WordPress backend could not be used in much the same way as CouchDB (given I do something like that with NarraFirma and WordPress myself). I also have my own experimental project from early last year (Pointrel20140331) that uses IndexedDB to support editing and running/rendering JavaScript and HTML snippets in a web browser, tracking all the changes; it too could benefit from having a WordPress backend to share data and be easier to find and install.

How might the polling issues you’ve thought about affect WordPress?

I’ve thought a bunch about the issue of polling and its limitations. One reason is that the shared hosting we’ve used for over a decade has a ban on chat programs because they can put a high load on servers. I expect many web hosts have similar policies to maintain quality of service for serving standard web pages.

A socket.io server (as mentioned above) is something Automattic might want to someday build into the WordPress core (somehow), as an option to go beyond the heartbeat idea and improve multi-user WordPress responsiveness (such as for plugins like NarraFirma).

But there is an alternative to trying to shoehorn WebSocket-related changes into the WordPress core PHP somehow. Such an update notification service to reduce polling could instead be hosted at WordPress.com in parallel to a WordPress server for a fee for those who don’t want to run Node.js (or whatever) locally to supply that service.

For an example of what Automattic does now in the area of parallel servers, when using shared hosting, people can still use “After the Deadline” servers at Automattic through Jetpack for grammar checking rather than set up their own. So, there is precedent for such servers.

Such a WordPress feature to reduce polling needs might then be used to add near-real-time chat to WordPress. This might reduce the otherwise increasing desire for Automatticians to use Slack to communicate with each other about WordPress. That is a movement that seems problematical to me concerning WordPress’s future. It would also be like Automattic moving to Pusher instead of improving WordPress to be low-latency and near-real-time. Even if Slack (by all reports) is a great product (as proprietary products go) that can monitor and log and search all of your communications stored on their servers, in theory using Slack widely within Automattic makes Automattic’s internal communications available to the same proprietary competitor (if even just rogue employees) for business intelligence purposes. It also seems strange to me for Automattic to cede the area of near-real-time communications to a proprietary company without at least trying (really hard) to come up with an alternative built into WordPress, especially because such things have already been shown to be workable. I feel Automattic could (and should) be innovating in this area to help the WordPress community move in new directions.

How would you implement something to reduce WordPress polling load?

I would probably enjoy working on adding such a feature of near-real-time chat via socket.io for the WordPress core. There could be two versions of such a service, perhaps called “PingPong” or “HeadsUp”.

One version of “PingPong” might just sent pings telling clients viewing a WordPress post or shared JavaScript-powered activity like an O2 chat to check for changes to a WordPress server via an always-fruitful “pong” poll for changes. The poll is always fruitful because the clients only poll when then PingPong server tells them to, although they might also do a fail-safe poll once a minute or so just-in-case. Each WordPress page (or Pointrel journal or topic) could have its own meaningful topic string, hash of a topic string (for privacy), or just random UUID to distinguish it from others as a unique “channel” for changes. Pings would be caused by either the client initiating them directly through WebSocket back to the PinPong server, or perhaps more reliably by the WordPress server telling the PingPong server when WordPress accepted a change like an updated post or new message adding a triple to a triplestore.

A single PingPong server could potentially support a large number of users looking at a large number of pages. Node.js would really shine in such an application given how it can handle thousands of active socket connection.

Still, to scale, there would eventually need to be a lot of such servers for load balancing, requiring some way to either specify to the client which server it should use or to route pings to a server through the network service’s front end load balancer. Balancing might be done by dispatching to the right server based on the first few letters of a hash of the topic name or random UUID.

That service would probably not require authentication, given the limited data transferred of just a message to check for updates. While no doubt there could be denial-of-service attacks by posting updates to random pages, a client-side rate limit of one poll per second could reduce the potential for mischief.

Another version of this service might require authentication and would pass messages with the actual incrementally changed data to reduce WordPress server polling load. For clients of this alternative version of the service, there would then be no need to retrieve related change data from the WordPress server except at startup or when changing pages, although any changes made would still be stored on a WordPress server.

That second authenticated version of the service could also supply user information for each change to make a “presence” indicator more efficient to do.

Both the unauthenticated low-information-notify-message service and the fully authenticated message-with-full-contents service could become extra Automattic services supplied either for free or at a fee. Keeping with the WordPress model, these servers (not much code involved anyway) should be open source so that individuals could also run their own without vendor lock-in (same as WordPress.com vs. WordPress.org). Lack of vendor lock-in makes relying on the convenience of the Automattic-hosted service a low-risk proposition.

Such a feature would enable all sorts of fine-grained interactions between WordPress users, such as interactive whiteboards, structured argument systems, near-real-time collective IBIS concept mapping, bbPress enhancements like better presence indicators, and more.

Combining near-real-time socket.io-based messaging in the WordPress core with also adding a triplestore in the WordPress core would open up even more possibilities for new JavaScript-based WordPress plugins using WordPress as a platform.

Perhaps such improvements, plus a bunch of new JavaScript-emphasizing plugins (like to turn WordPress into an even better co-evolving thinking-together tool like Doug Engelbart talked about) might even get WordPress another 25% of website market share? :-)

Is this all talk and no action on the polling-elimination server for WordPress?

I’m 95%+ sure this would not be that hard to do, and would not have significant cross-origin issues, at least as far as the version that just sent pings and no data. I’d love to work on such a thing as I sketched out above, and I don’t think it would take that long to implement.

A proof-of-concept demo of such a “PingPong” or “HeadsUp” system could involve making (or adapting) a JavaScript single page web app supporting near-real-time chat and/or a whiteboard as a WordPress plugin. Eliminating polling would require implementing and then setting up an Automattic-hosted Node.js socket.io server that just relays “heads up” pings without any data (to reduce security issues) to all browsers viewing the same WordPress page (which “channels” distinguished by a random number associated with the page). When the browsers get the ping via socket.io, they can request new data from the WordPress site without needing to poll for changes. None of these are that hard to do in theory — especially as I’ve essentially done each part locally in practice separately for NarraFirma and TanksInYourBrowser. :-)

Such a proof-of-concept might be a great audition/trial project for me at Automattic… :-)


How does NarraFirma fit into a bigger picture of public intelligence and sensemaking decision support tools?


tl;dr There is a bigger picture NarraFirma fits into:

  • NarraFirma is is an example of a public intelligence and sensemaking decision support tool
  • There is a broader “Twirlip” concept that NarraFirma fits into
  • WordPress is a great platform for deploying public intelligence and sensemaking decision support tools
  • Automattic could consider increasing its investment in decision support tools to grow the WordPress ecosystem — like by hiring me to help with that as a Janitor :-)

What do you mean by “sensemaking” and “intelligence”, and how do they relate?

Sensemaking is “the process by which people give meaning to experience”. It often has an emotional aspect and is often a holistic “right brain” activity, yet it also has analytical “left brain” aspects.

Intelligence, as I mean it here in the “intelligence agency” sense is “the collection, analysis, and exploitation of information and intelligence in support of [various] objectives”. Intelligence tends to be more an analytical left brain activity, but is also has holistic right brain aspects.

Both sensemaking and intelligence are complimentary to each other. Both can encompass each other to some degree. Both are important to a society.

A lot of sensemaking is done informally, day to day. When sensemaking is done more formally, it is often one aspect of larger “intelligence” process (although sensemaking also goes on separately from intelligence work).

For that reason, I still tend to lump sensemaking under intelligence when I talk about “public intelligence”, but as above, the reality is more nuanced. Sometimes I use a much longer phrase of “public intelligence and sensemaking” to be more precise about the overlap.

It may be tempting to think of both intelligence and sensemaking as ways of creating “summaries” or “presentations”, as those are frequent end results. However, to an extent, both intelligence and sensemaking work (sensemaking especially) are also the opposite of summarization or presentation, because they are often deep explorations into uncharted territory often involving participation in communities of people with the same or different interests. One result of such a process may be a summary for a decision maker and a a list of options and likely consequences as part of a presentation highlighting important trends with easy to understand visualizations — but behind such a result may be a vast amount of computer-assisted discussion and learning without which a quality result would have been impossible.

As an example, my own tagline I use in my email signature of “The biggest challenge of the 21st century is the irony of technologies of abundance in the hands of those still thinking in terms of scarcity” is in a sense a summary. (People reading this document may well believe I am otherwise incapable of conciseness. :-) However, that summary is also the result of a sensemaking journey of decades both alone and as part of various web communities like Slashdot and various mailing lists (Space Studies Institute, Princeton’s “Advocates & Skeptics”, Michel Bauwen’s Peer-To-Peer Research, OpenVirgle, Open Manufacturing, etc.) without which that summary might not have been possible. And that overall experience informs a lot more personal decision making (including writing this document) than just passing on the summary by itself.

To the extent decision makers may be removed from the sensemaking process, summaries may be essential to making the best decisions possible in limited time (like for the President of the USA who is a very busy person compared to thousands of intelligence analysts working on Presidential briefings at the CIA, NSA, FBI and so on who collectively in theory have much more time for reflection and exploration). To the extent decision makers are deeply involved in the sensemaking process themselves, brief summaries may seem laughable and long-term commitments based on them may be seen as likely to lead to disaster (especially when decisions may need to be continually remade hour-by-hour as circumstances change based on previous actions in an OODA loop or similar). Hopefully, citizens in a democracy deciding how to vote, whether to run for office, or what to suggest at town meetings and so on, may move more often into that latter category using WordPress-powered intelligence and sensemaking tools.

A summary like my tagline is itself in a way a poem or index pointer to deeper layers of explanation (which themselves link in turn to even more onion-like layers of poetic meanings built with yet more pointers and relationships).

Summaries and presentations are important in many situations, and working towards them can be an important way to reflect on priorities, but summaries are more like temporary expedient rest stops than the long ongoing sensemaking journey itself.

What is public intelligence?

Public intelligence uses “open source” (in the intelligence sense) data to try to understand opportunities, risks, and trends, and it does so in a public way generally open to outside commentary.

Since things are rarely 100% pure, often there may be a mix of some private or not well known information mixed in explicitly or implicitly in any intelligence work. So, in practice there is a spectrum from a lot of openness to a lot of secrecy. Any real activity falls somewhere on this continuum, perhaps even in multiple places at different times depending on the phase of the intelligence work or the focus of the intelligence work. Groups often cooperate in one way and then compete in another; one term for that is “coopetition“.

Here is a contrived example of a mixed situation. A business that makes desktop computers might join a trade working group with public meetings archived on the web and openly discuss computer industry trends based on published data as part of a collaborative process with other businesses. That working group might use open source public intelligence and sensemaking decision support tools that help make sense of a large amount of readily public data ranging from government census statistics to published past industry sales figures to web search results on recent high profile commentaries by industry leaders and highly followed pundits. That group might consider public comments made by a variety of individuals who pay attention to the proceedings. That group might publicly conclude, among other things: “Computer desktop sales are declining, and it looks like there is nothing anybody can do about it.” Then, confident in that that trend based on the working groups extensive research, the specific business’ staff might develop their own in-house plans based on not-well-known local knowledge of their own capacities. The staff might even use the same open source tools the public working group used, but with private data. The staff might privately conclude: “Because we know a lot about making metal boxes and just invested in a huge sheet metal press, we could move into supplying automotive manufacturers with custom sheet metal parts and undercut the competition there.”

For excellent and inspiring background reading on open source public intelligence work, see Robert Steele’s book “The Open-Source Everything Manifesto: Transparency, Truth, and Trust“. Among many great points, he write about the importance of bloggers at citizen “intelligence minutemen” determining “true costs” for things and keeping government accountable to the People’s needs in a very fine-grained way. He goes into detail on one approach to do that in the book. Here is a related talk he gave with some slides. I’d like to implement as experiments many of the ideas he outlines, perhaps even (with permission) using some of the charts and tables he has in the book as the first versions of GUIs for drilling down into issues and accessing related tooling for data entry, structured arguments, multi-perspective sensemaking, and so on. Likely over time any community will together incrementally build something better than what any single individual can think of (whether Robert Steele, me, or anyone else), but he has already thought deeply about these issues for decades, and so his work provides a great starting point as a requirements specification for implementation work. There are also other more specific sources of information on sensemaking and intelligence techniques which I mention later; any such techniques could fit within the broad public intelligence framework Robert Steele presents.

By the way, I do agree 100% with everything Robert Steel says politically or how he frames all of the issues. Nor I expect would he likely agree with everything I might advocate. However, agreeing politically is besides the point when people choose to collaborate together on building and using a system for “democratic” discussion and decision making (or whatever other participatory forms are supported like “chaordic”: or “stigmergic” or whatever). It is the very purpose of such systems to help work through political disagreements in respectful productive ways. Building an IBIS concept map together is one example where people opposed on various issues can work together to at least build a complete map of the political landscape, even if the journey a community takes across that landscape may still be yet to be decided. So too for working together to build even just a list of pros and cons about an issue, when areas of uncertainty or disagreement can be flagged. This TED talk by Michael Sandel on “The lost art of democratic debate” also makes the point about working together despite differences in a democracy. There is even an entire theory that human’s abilities to reason and argue evolved not so a lone individual could find the “truth” but that so a tribe of individuals could work together from various perspectives and experiences to arrive at actionable plans that were based on a best approximation of the truth as arrived at collectively — precisely by people disagreeing with each other in front of an audience.

So, an important aspect of public intelligence work is that people can use web-based “decision support” tools to work together in a stigmergic way to construct a large artifact together about political situations — including by supporting flagging areas of agreement and disagreement. Users of such a system may then tend to filter the results based on their perspectives and priorities, but at least all the possibilities have been outlined comprehensively. As feedback from the results of choices continues to feed into the system, and also as users, information sources, decision making processes, and tools all develop reputations in various ways in different contexts, the overall process can continue to improve.

This is similar to how, in theory, a deliberative legislative body composed of many different elected officials is supposed to work collaboratively through public discussion relating knowledge to the needs of the People — and then engage in consensus building (or, voting) towards plans of action to address perceived needs. That process may have become dysfunctional in many legislatures for whatever reasons, but one can hope the public intelligence tools can contribute to its renewal.

What are decision support tools?

A key term Robert Steel uses is “decision support tools”. Decision support tools are a big part what I am talking about creating as part of “Twirlip” or whatever such a project might be called at Automattic. Public intelligence involves the application of such decision support tools to public issues in public ways using “open source” data. As in the above contrived example, such decision support tools can also be used for private intelligence just within an organization as well as with “private” data.

Viewed broadly, there are many existing decision support tools ranging from spreadsheets to collaborative P2 blogs with priority lists to Compendium to SEAS to Angler and beyond. So, a lot of what I’m talking about implementing for Automattic and the WordPress community is not new. I’m mainly proposing bringing variants of such tools directly into the WordPress ecosystem as FOSS plugins — just like my wife and I did already with NarraFirma. Ideally such tools would use a common standards for semantic data exchange (like via semantic triples) so that they can more easily interoperate; however, given the interlinked nature of the web, such tools are useful even if they otherwise don’t directly interoperate. Obviously, a good first step before writing anything new is to inventory what is already out there as far as WordPress plugins for decision support, and then to likewise inventory more of what decision support software is out there that is not already a plugin.

To get a sense of what general areas such decision tools might cover, see the “Figure 35: Generic All Source Information Handling Functionalities” from page 136 of Robert Steele’s “The New Craft of Intelligence: Personal, Public, & Political; Citizen’s Action Handbook for Fighting Terrorism, Genocide, Disease, Toxic Bombs, and Corruption”. According to that book, the diagram is based on the work of Diane Webb under the oversight of Gordon Oehler in the Office of Scientific and Weapons Research (OSWR) at the CIA. That diagram can also be found in this 2012 presentation on “CATALYST Revisited: Challenges to Creating Advanced Analytic Environments” by Diane Webb (now President of BizInt Solutions). A later revised version of the diagram can be found online here in part IV of “Intelligence for Earth – Clarity, Diversity, Integrity, & Sustainability” by Robert Steele (although with some parts omitted). The diagram shows the flow of information from Open Literature, Non-Text Data, and Restricted Information to eventually (after several layers) Finished Intelligence and Reporting. As with Robert Steele’s later diagram, people can approach any module in the system in various ways, so the whole concept is more interactive than a straight waterfall model. The modules are in three broad layers moving from acquired data to polished reports, each layer itself having two sublayers.

Here are the key areas Diane Webb suggests and which all could be added as “decision support” plugins to WordPress (for capabilities that are not already there) to support public intelligence and sensemaking, from bottom to top:

  • Input: Open Literature, Non-Text Data, and Restricted Information
  • Layer C2
    • Conversion of paper documents
    • Automated foreign language translation
    • Processing images, video, audio, and signal data
    • Automated extraction of data elements form text and images
    • Standardizing and converting data formats
  • Layer C1
    • Clustering and linking of related data
    • Statistical analysis to reveal anomalies
    • Detecting of changing trends
    • Detection of alert situations
  • Layer B2
    • Interactive search and retrieval of data
    • Graphic and map-based visualization of data
    • Modeling and simulations
  • Layer B1
    • Collaborative work
    • Notetaking and organizing
    • Structured argument analysis
  • Layer A2
    • Desktop publishing and word processing
    • Production of graphics, videos, and online briefings
  • Layer A1
    • Revision tracking and realtime group review
  • Output: Finished Intelligence and Reporting

Also, I would tend to add “sensemaking” support in to that list somewhere, although such tools might span multiple layers like NarraFirma does.

If you squint at this list, you can kind of see much of Google’s product line and R&D efforts. :-) But Google is not focused on empowering the individual and improving the web through decentralized democratic publishing and dialog, compared to Automattic. Google’s business model, built around renting participant eyeballs to advertisers, is somewhat incompatible with decentralized use and local information storage such as WordPress supports (even if it may be more convenient and cost-effective to use WordPress.com for many people). Google also has moved more and more into proprietary software and effective vendor lockin. Even in past April Fools jokes like Project Virgle from 2008, Google does not quite get post-scarcity possibilities. So, I’d rather help Automattic than Google, even though I could in theory otherwise do the same sorts of general public intelligence work at Google in a more centralized way. Probably such work could even be done more easily and with better immediate community acceptance (without an uphill struggle, including writing a document section like this one) at Google rather than at Automattic given Google’s long-standing emphasis on organizing the world’s information (although I would not go so far as Alex Jones’ claims about direct broad CIA support to Google from the start).

Making WordPress into a public intelligence platform is no doubt a tougher “sell” to the WordPress community. Many WordPress community members may rightfully recoil at a lot that parts of the intelligence community have done in their name, actions which have lead to all kinds of blowback and disaster both inside the USA and globally. The word “intelligence” is in that sense deeply tainted.

Still, Automattic’s broad mission of making the web a better place is none-the-less potentially compatible with public intelligence and sensemaking tools. Automattic might even be able to help reform the connotations of the word “intelligence”, perhaps to include tools for supporting both logical intelligence and emotional intelligence (working together through social intelligence)? Many tools have multiple uses. NarraFirma could be used to help increase recruitment and retention for any organization (good or bad), just like WordPress could be used to spread either loving ideas or hate speech. Langdon Winner points out how things are more complex than that given complex dynamics of infrastructures and social processes, but we can still make choices in how we use our tools. For good or bad, governments have intelligence and sensemaking tools. The question is more, should the People have such decision support tools too? And if so, is WordPress a good vehicle for delivering them?

WordPress already helps a lot in layers B1, A2, and A1 in that list above, although it could do more like with SEAS-like structures arguments in B1. A prototype “Concept Demonstrator” my wife and I developed for Singapore’s Horizon Scanning and Risk Assessment (RAHS) system in the mid-2000s covered several aspects of layers C1 through A2, and they are not that hard to provide basic coverage for. NarraFirma adds coverage to WordPress at the C1 level already and some other B and A layers within a narrow range of focus (dealing with raw real-life experiential stories).

As the F18 group at the US General Services Administration (GSA) suggests in “How to use more open source in your next federal IT [acquisition]”, software tends to move from an idea in a specific place an time (like Codd and the relational model in 1970), to a specialized in-house implementation (e.g. custom DB software like at various big companies), to proprietary COTS (e.g. PostgreSQL originally in 1985, MySQL in 1995, or Oracle then and now), to specific FOSS products (e.g. PostgreSQL in 1996, MySQL in 2000), to broad community support and ease-of-use (PostgreSQL now, MySQL/MariaDB now). Publishing has followed a similar path, leading up to the WordPress community now. We will likely see the same trend play out broadly in intelligence and sensemaking decision support software too, as such software has multiple uses beside national security as Robert Steele explains in detail. The FOSS NarraFirma software which my wife and I wrote is itself is an example of that, as it was preceded by the COTS Sensemaker Explorer (which my wife and I also wrote much of), to RAHS (again, we helped the core of that), and before that CATALYST that Diane Webb talks about.

That intelligence system history goes back even further though, indirectly, to Minitel, Dynabook, Augment, Cybersyn in Chile (destroyed after a US-supported coup), PLATO, and earlier electro-mechanical/photographic ideas go all the way back to MEMEX by Vannevar Bush. That history goes back even earlier via direct writing and human filing and retrieval using indexed notecards in 1930s-1940s France (destroyed by the Nazis) to the first libraries with clay tablets and back even back to the first instructive break-through “Standing Bear” cave drawings Paula Underwood wrote about in The Walking People. Sensemaking systems go way back as well to the Tarot, the I Ching, the Medicine Wheel, and more.

So, there is nothing new about public intelligence and sensemaking, even as the tools or how they are developed, supported, and used may change. Bringing more such ideas to WordPress just feels to me like a natural incremental evolutionary step, given WordPress supports about 25% of the web and Automattic has aspirations to grow that percentage further.

What about Wikipedia and public intelligence?

Wikipedia in theory might have become such a public intelligence tool (and still might). However, Wikipedia focuses on constructing a single authoritative encyclopedia-like article on a topic with a neutral point of view citing secondary third-party sources with established reputations. Such articles can indeed be valuable things; I link to several Wikipedia articles in this document for example. I’m not suggesting Wikipedia change its focus to that of, say, everything2 (or, for that matter, WordPress.com). Wikipedia is what it is. But an authoritative article is only one of many fruits of a public intelligence garden which would support a variety of approaches, perspectives, and outputs. And then there is also the way you go about producing such articles in any case in terms of tooling used.

An encyclopedic approach to knowledge is in a way fundamentally in opposition to supporting a diversity of voices who each have a coherent perspective derived from a variety of personal experiences as well as from reading a wide variety of sources and thinking creatively about a topic (often arcane topics of little general interest). Those sorts of things are generally big “no nos” on Wikipedia (like no original research, no non-notable topics, no non-neutral point of view articles, and so on).

Google’s now defunct “Knol” project was a step to address that by supporting a variety of articles on a single topic, however who needs Knol if you have a network of WordPress sites? :-)

Also, for either Wikipedia or Knol (or WordPress), there are no tools for things like structured arguments, concept maps, more formal sensemaking, and so on to easily ride on top of the collected raw material. Wikipedia struggles for lack of such tooling IMHO even just in service of its primary encyclopedic mission. And in part because of that lack of tooling and supporting of biased reporting (flagable as biased from some other perspective of course), Wikipedia struggles with editor burnout, like when editors construct coherent (to them) articles only to see them trashed. That burnout then spills over to admins. While I do hear of bloggers who move on to other things, I rarely hear of them burning out in the way Wikipedia editors and admins do.

Wikipedia also suffers from various non-obvious biases of its own. One example is that almost all the medical information is mainstream, and information about alternatives tend to be systematically repressed. It’s certainly great to have a good site for mainstream medical information when you want that; my point is that the conflicts and limitations involved in such choices are not readily obvious or easily explorable with the current Wikipedia approach whatever one thinks about mainstream alternatives. Wikipedia presents one current authoritative-seeming face to the world, not a network of possibilities and perspectives. You can dig through old article revisions on Wikipedia to find alternative ideas, but that is cumbersome.

The closest aspect to “public intelligence” work in Wikipedia is probably not the “Article” page so much as the “Talk” page. Talk pages involve discussing the value of different sources, and the clarity of different phrasings, and also the nature of various biases by various authors or supplied text. These are all good things to do. However, complex discussions about articles by multiple individuals with different perspectives become hard to hold in one page of wiki text without better tooling. There is no easy way to just flag an argument (flagged by some individual from some perspective for some purpose) in some machine-readable sortable way but leave the argument in place alongside others. The same is true for WordPress comments, of course. Wikipedia has been working towards better tools for discussions, so we will see where that goes. It is possible that any JavaScript-powered WordPress plugins for public intelligence might be readily adaptable for use on Wikipedia talk pages or vice-versa.

How is public intelligence different from non-public intelligence?

To understand “public intelligence” it is first important to consider regular non-public “intelligence” for national security and business and how that process has typically operated up until now. Secretive intelligence work generally relies on non-public sources such as typically done by groups like the CIA, the NSA, the DIA, the (old) KGB, MI6, etc.. For example, here is the mission statement for the US CIA: “Preempt threats and further US national security objectives by collecting intelligence that matters, producing objective all-source analysis, conducting effective covert action as directed by the President, and safeguarding the secrets that help keep our Nation safe.” Such organizations like the CIA tend to culturally prioritize and reward collecting “secrets” (or in some cases “covert” spreading of misinformation) — even though such organizations also collect a lot of public information (“all-source”), try to make sense of it, and will also often communicate correct information as well depending on the circumstances.

Such secretive intelligence organizations are often ineffective in many ways due to “siloing” of information both for technical reasons and due to internal political infighting over turf (common to most large bureaucracies). A general air of secrecy and “need to know” policies often prevents information from circulating to where it needs to go to deal with some emerging crisis. However, such “silos” seem almost inherent to a certain culture that hires and rewards for dealing in secrets. One might even suggest that, outside of investing in data collection technology, things have progressed to the point Robert Gates warned about in 1995 in this officially released CIA document: :-)

What we need to do is to squeeze more relevance, more value, and more information from the existing resources the Community has. We have to fix the bureaucratic legacies of the Cold War. We have to invest in the future in collection technologies, in information processing, in language training, and in a variety of other areas. The Intelligence Community has to come to grips with the reality that, while preserving its role and missions and capabilities, it does not have to preserve the current way of doing business. … And if the Intelligence Community doesn’t have the boldness and imagination to figure out a way to come back to the commission with some useful ideas and say, “now here’s how we can do what you want to have done,” then you’re going to have to take your chances with a group of people who don’t know how to do it. [Links added to make a humorous point. :-) ]

Analogous to intelligence agencies dealing in a lot of secret information, big businesses may also collect “proprietary” or “trade secret” information and then make decisions based on it, doing what could be termed “private” intelligence work. “Proprietary” or “confidential” in that context is often used to mean “secret”. For example “IBM Confidential” communications tagged in Lotus Notes are meant to be kept secret within IBM on a need-to-know basis or only otherwise be shared with other organizations when specific confidentiality agreements are in place to continue to guard the secret. These secrets may be specific data that companies have collected by paying for specific confidential reports. Or, companies like Google, Slack, GitHub, Twitter, or similar services can potentially data mine streams of third-party data they have access to related to services they provide. They may aggregate such information to produce services like Google Trends or targeted advertising as with gmail. In theory, they might use such data for internal decision making as well (subject to possible terms of service legal constraints). Even just metadata about who is talking to whom about what subjects how often (such as Google potentially has ready access to via gmail headers) could provide valuable and actionable information about business investment strategies (in the same way national intelligence agencies use that in outlining military strategies). Often these company secrets may be about in-progress plans not yet implemented or in-progress designs not yet for sale. However, such secrecy can also be used for estimates or projections based on public data — key information behind plans and designs. Any big corporation, especially a publicly traded one, tends to have lots of internal secrets of various sorts (including usually salary data). The release of corporate secrets especially relating to financial issues can be a complex process as they may affect stock prices and legal issues may apply such as insider training rules. So, every secret is a potential risk to be managed in that sense.

Because secrets incur various costs (as well as potential benefits), the question is, what secrets are really worth keeping and for how long? And what should an organizations “default” mode of operation be when there is not a compelling reason to be one way or the other — closed/secret or open/public?

As ex-CIA officer Robert Steele suggests, intelligence processes go on in multiple sectors of society, not just secretive organizations. Robert Steele talks about several “tribes” of intelligence (including media, business, military, law enforcement, government, health, academic, non-profit, and civil society). Each “tribe” has its own unique challenges and obstacles to sensemaking and intelligence works. Still, there are broad overlaps in the needs of each group.

What I mean by “public intelligence” is the “public” version of the “intelligence” process. By contrast with intelligence agencies and confidential aspects of corporate governance, public intelligence emphasizes using publicly available data, where that assembled data is then reflected on and discussed in public ways. As an analogy, this distinction is akin to the one between “free and open source” software development processes and “proprietary” software development processes.

Since it is rare that a process is 100% pure, I’m talking about a general emphasis here on sharing information and thinking through things cooperatively, not complete absolutism (especially within today’s economic system). For example, a person can participate extensively in a public mailing list exploring a topic while still having a few private side emails. That person may be doing a lot of important public intelligence work even without being 100% public with everything.

What is the potential scale of public intelligence work?

The potential scale of public intelligence work ranges from local neighborhoods (such as understanding local historical employment trends done by a town supervisor) up through larger states (like tracking statewide crime trends in relation to leaded gasoline exposure done by a legislative research service) and onto the entire planet (such as climate change modeling done by a network of scientists). While the scale of such involvement may typically track the scale of the issue, there is no reason a network of scientists can’t look together at the specific challenges of a small town, or a town supervisor can’t think deeply about global climate change.

Another way to get a sense of scale on what is possible is to consider the growth of the NCDD over the past decade:

“The National Coalition for Dialogue & Deliberation (NCDD) is a network of more than 2,200 innovators who bring people together across divides to discuss, decide, and take action together effectively on today’s toughest issues. NCDD serves as a gathering place, a resource center, a news source, and a facilitative leader for this vital community of practice.

Dialogue and deliberation are innovative processes that help people come together across differences to tackle our most challenging problems. In a time of extreme political partisanship and increased conflict between religious and ethnic groups, teaching, spreading, and supporting the skills of dialogue and deliberation are vital.

The NCDD website is a clearinghouse for literally thousands of resources and best practices, and our highly participatory national and regional conferences have brought together more nearly 3,000 practitioners, community leaders, public administrators, researchers, activists, teachers and students since 2002. We keep in touch monthly with 34,000 people involved in public engagement and conflict resolution work.”

While some of what NCDD practitioners do may be proprietary such as to resolve issues within a conflicted company, the fact that there are thousands of practitioners working in this area shows the potential demand for better tools to support such activities.

A key part of the intelligence process is reconciling different perspectives from different analysts; even though these analysts may be in the same organization, this process is potentially not all that different from helping bridge divides between different groups in different organizations. When “public intelligence” tools are open source, they can be used in all sorts of ways, including for “private” intelligence such as resolving private internal disputes within an organization.

Where has such public intelligence work been going on in the past?

Academia has for decades been a place where public intelligence is done, especially since rewards and prestige are based on publishing. (Although, it has only been more recently that actual study data from academia is more commonly openly distributed.) Likewise, newspapers have historically also filled that public intelligence role to some extend (back when newspapers were in less financial distress and more “investigative” reporters actually did more investigating than just reading press releases). Conventional independent journalism can be seen as a form of public intelligence work, as a journalist collects public data (or data someone wants to make public) and then typically tries to fit that data into a narrative on some issue. There also are many non-profits and foundations of all sizes who focus on specific issues (including small groups of concerned citizens who may meet informally). These non-profits may prepare related reports or network with other non-profits with overlapping goals to do education and advocacy. Industrial trade organizations are another place such work may go on. And, of course, legislatures have long engaged in this sort of work, with public deliberations in Congress down to the smallest town board.

To a lesser extent, as a sort of edge case, big businesses such as IBM are another place public intelligence work is done on a large scale as far as using public data to make sense of things, even if the process is not public and the results may not be made public for a long time. Big businesses generally need to understand what has been going on in some industry, what is going on right now, and what may be going on soon (especially to understand market trends to see business opportunities). That corporate work is public intelligence work in the sense that the businesses are generally working from public data, even if the people doing the work are mostly employees covered by confidentiality agreements and the results are often not shared publicly right away, although the results may sometimes be reflected in required corporate quarterly reports, annual reports, IPO documents, or other shareholder reports.

How is public intelligence work changing now?

Even within the world of business and government things have been changing towards openness. As a contrast to the way most businesses operate, some companies following “Honest Business” strategy advocated by Michael Phillips (a founder of MasterCard) have long opened their books (especially around San Francisco, inspired in part by the Whole Earth Catalog). Some computer companies have adopted similar policies. From a public policy standpoint in relation to corporate limited liability, one might even question whether powerful corporations legally should be able to have secrets and have a corporate right to privacy in that sense (as a recent US Supreme Court decision shows). And in practice, corporate secrets are getting harder to keep in the internet age (Glassdoor being one example). Government bodies like state legislatures (the New York Senate is a great example) are also increasingly going open as the open government movement grows and grows and grows. Former employees at intelligence agencies are calling for more openness as well. DARPA is wondering if transparent computing might thwart cyberattacks. Bridgewater (a hedge fund) is an interesting example of an organization pushing for “radical transparency“, but only inside itself. The FBI is likely adopting some of those policies too (inspired by Bridgewater’s approach). So, there is likely to be an increase in demand for new tools that are based around openness (although there may still be in-group/out-group community-based boundaries to that openness).

The internet makes possible “crowdsourcing” of public intelligence work at low cost. Cheap computers with big hard drives make possible collecting and locally analyzing a huge amount of data by yesterday’s standards. The “cloud” (i.e. renting somebody else’s cheap computers) opens up even more opportunities. Using the internet, a rural town supervisor in New York could collaborate with town supervisors across the USA to think together reasoning from first-hand experience about how various national trends are going to affect rural populations. That sort of collaboration was much more expensive to do years ago as it required travel, expensive long distance phone calls, photocopying costs, buying filing cabinets and so on, and was limited by the ability of individuals to collect, sort, and chart data by hand.

What do you want to do related to public intelligence?

While Cynthia reads widely in areas including narrative, sensemaking, and public intelligence and has various interests (even making related games to promote story sharing), her focus so far as a consultant for hire is on using and promoting the specific participatory narrative inquiry process she has invested in via her book and software designs.

My own interests are a bit broader though (going all the way back to the Pointrel triplestore in college). I would like to help bring all these sorts of public intelligence processes to the web, and I see NarraFirma as a first step towards doing that.

A variety of people are doing bits and pieces in this public intelligence area. Douglas Engelbart in particular worked for decades to help people do this sort of stuff. But there is still a great need. How can we address that need for better public intelligence and sensemaking tools?

What if a group of people formed a “public intelligence working group” and started by taking a good look at what tax-funded intelligence agencies actually do internally — as well as what such places would like to do? To be clear, that question is meant as looking at the parts of the intelligence agency process that are publicly known; I’m not interested in obtaining “secrets” given there is so much public data is out there.

Like walking around in a big city, go down the wrong down alley in the intelligence community and you are probably at risk of getting mugged for whatever reasons. I’m just talking about staying within the brightly-lit and well-documented main avenues of intelligence work For example, as a starting point, the CIA has a page on “Intelligence Literature: Suggested Reading List”. Similarly, the Wikipedia article on “Intelligence analysis” is another starting point. Also, Eric Patton has put together a list of various existing tools one could learn from or use for open source intelligence. While the specifics of what national security intelligence analysts work on may be classified, the general techniques they use are common knowledge. As with the CIA, such organizations can even see getting more people to know about what they do (up to a point) as a form of recruitment and/or public relations like on the CIA Kids’ Zone, the same as how the US Army sponsors a video game.

And what if this public intelligence working group then tried to bring the best of those ideas to the web and make them available to everyone for free? These are ideas such as have been embodied before in SEAS, ANGLER, ANALYST, and RAHS. There are also variety of other ideas that connect more indirectly to intelligence work like IBIS and are embodied in Compendium (and which intelligence analysts probably could be using even if they might not be). These ideas can be learned from the public literature, reviewing public press releases and web pages, and also perhaps even from Freedom Of Information (FOI) requests related to defense contract specifications and such.

Such a group obviously might not want to do everything “TIA” aspired to do (nor should it or could it), but some parts of the overall intelligence agenda, like Genoa II, are well worth studying and building on. NarraFirma itself was built on ideas whose research was supported by the Genoa II project and the late Tom Armour. Thanks, Tom, for your years of service to your nation and the world.

As Tom Armour wrote:

Genoa II’s predecessor program, Genoa, was about getting “smarter” results by harnessing the diversity of lone rangers, bringing them together as a team and supporting them with technology to discover relevant information, reason systematically about it, and capture and reuse knowledge created by this and other teams. We believe that Genoa I did produce better, deeper understandings of complex situations, but it did so at a price: speed. Lone rangers are, after all, much more nimble that most teams.

With Genoa II, we want to improve on both dimensions, and do so with teams that function at the edges of existing organizations while having access to the information and other resources of the participating organizations. I will talk about the three themes of the Genoa II program. These themes are: becoming faster through automation; becoming smarter through what we are calling cognitive amplification; and working more jointly through center-edge collaborative technology.

Also, again like any big city, any big organization like the three letter agencies are full of different people with different perceptions and responsibilities. There might even be people in such agencies at some point whose job it is to help with something like public intelligence, even if some other parts of the agency might have different opinions. For example, one part of the US NSA created Security-Enhanced Linux under the GPL and Red Hat now provides commercial support for it — even as potentially another part of the NSA might want a backdoor into Linux. It is not unusual for big organizations to have conflicting priorities within themselves. In fact, the frequency of that occurrence is one reason why sensemaking and public intelligence tools to consider multiple perspectives and peacefully resolve conflicts are so important, and why the work done by NCDD practitioners is so important.

One could see the current network of millions of “lone ranger” WordPress blogs as a crowdsourced public intelligence network of a sort — even though WordPress was not designed specifically to be a public intelligence platform.

WordPress has bits and pieces of some of the technology of connecting teams of solitary bloggers (including pingbacks). And of course P2/O2 provide support for teams collaborating by discussions. Still, using WordPress for public intelligence work via just making posts and getting comments is probably kind of like doing math calculations with pencil and paper instead of a calculator or spreadsheets. You can do it. Some few people can do brilliant stuff by just thinking and making a few notes. Pencil and paper may even be the best thing to use for a lot of thinking. But calculators and spreadsheets are often used by people for reasons like speed, accuracy, and scale.

For just the smallest example, in the case of WordPress and public intelligence, just to start, where is the easy way to quickly flag comments for logical fallacies (ad hominem, overgeneralization, etc.)? How do you note the type of supporting evidence for a position (first hand, second hand, etc.)? How do you sort comments on such things? How do you check for when posts do not have multiple rival hypothesis for a listed issue? Yes, you can do that all by hand via ad hoc text and reading a lot, especially if you already know various techniques of clarifying thinking (like a good English teacher might), but isn’t that sort of work something that computers and good UX could make easier?

The same is true for PNI without NarraFirma. You could stare all day at all the stories and metadata printed on paper and maybe make some graphs from them eventually using markers on a whiteboard. That might even be a good thing to do sometimes (including in certain workshop contexts). But using the NarraFirma software (or something similar) makes some parts of the sensemaking process faster and more accurate like systematically considering graphs for every combination of two questions, which ultimately lets you consider more stories in the sensemaking process and produce better results.

What if someone put some thought and effort into helping the WordPress network support that public intelligence task even better? What if ideas from Genoa I and Genoa II (and other sources) could be brought to WordPress to support teams of lone rangers working together?

Mr. Furious: Looks like tonight the lone wolf rides… alone.

The Sphinx: You must be like wolf pack, not six-pack.

What would all these ideas mean if thought through deeply in a WordPress context? I don’t know the answer to that, but I feel it is a question worth exploring.

What about finding investors for the Twirlip public intelligence platform?

So, rather than just focusing on improving NarraFirma, we (Cynthia and I) could alternatively try to go really big picture with the idea of creating a platform I call “Twirlip” as some kind of comprehensive information management and public intelligence platform and “social semantic desktop” for the web. Then we could try to get investors for that. NarraFirma can be seen as just the first tool in a growing “Twirlip” toolbox.

There are certainly companies that are funded in various smaller niches like just business intelligence (such as Looker or Tableau or more, even EverNote to an extent) — even if they are mostly not open source as web services and they generally require importing your data into their servers. So, money has been invested there before. And it is not like we have zero credibility in that space — after all, we earlier wrote the first version of the core of the software that an entire country uses for risk assessment and horizon scanning — even if it is a tiny country in total land area, but you have to start somewhere. :-) And now we have written NarraFirma.

I wrote “we” above, however as mentioned earlier, Cynthia (probably both from being more practical and being more experienced in the field) is more focused on just NarraFirma. I am more the one pushing to expand the toolset way beyond that into something called “Twirlip” for all sorts of public intelligence. She knows that just NarraFirma by itself could do a lot of good, and it is a fair point to suggest trying to do too much may lead to nothing getting done at all.

However, I still feel that it might be possible to get thousands of people involved in something like Twirlip eventually, all working towards better public intelligence tools. Something like Twirlip may well happen on its own, but I feel it would be “hard fun” and worthwhile to be part of that.

So, let’s say I wanted to get investors for Twirlip. I would want to stay open source with Twirlip, as well as to be very broad (as with a generic triplestore) and expandable via plugins while emphasizing support for small groups, as well as to support people keeping local copies of their data. Kind of just like, drumbeat, WordPress. Well, WordPress does not have a built-in triplestore, but NarraFirma/Pointrel starts to fix that. And probably there would probably eventually need to be ways to interface to big data, but WordPress could do that given the nature of the web, even just by embedding iframes.

But how to persuade investors to invest in such an idea? An investor-funded Twirlip would likely have to use the business model of selling back end server capacity and support — kind of like that company called … what is it that name … it’s on the tip of my tongue … ah, yes … “Automattic”. :-)

Eventually, if wildly successful against the odds, an investor-funded Twirlip would then end up competing with Automattic and WordPress as the platform for the web (even if Twirlip just ran on top of WordPress). After all, publishing information is in a way a subset of sensemaking and public intelligence. Once people see what might be possible with such tools, maybe everyone will want to use them for all sorts of purposes from personal financial planning to analyzing the global economy, from deciding how to plan a personal garden to predicting trends on global agriculture given climate changes. Or, to be clearer, people would be discussing models and resulting predictions about things like global climate change in Twirlip (not running the actual models, which typically takes supercomputers). Fantasy, maybe, but look at where WordPress was a dozen years ago compare to competitors like Movable Type, so the rise of Twirlip to turn the web into an effective public intelligence system over the next decade or two is not 100% impossible even if extremely highly unlikely…

But we all know competing with Automattic would be a dumb idea, right? :-)

Besides, even if it wasn’t a dumb idea to compete with Automattic, why not just do a friendly “takeover” or “merger” with Automattic instead? :-) Then I could just let Matt Mullenweg and the rest of the current Automattic team keep doing all the hard work they are already doing of running things their own way (why mess with a good thing?) so we can keep bringing in lots of revenue while supporting millions of users and also keeping the investors happy through continued growth. Meanwhile, with those time-consuming-but-important management, leadership, investor relations, community relations, PHP support coding, legal, payroll, happiness, advertising, and hardware management tasks delegated and off of my mind, :-) then I would get to have fun making great JavaScript-based tools to help the rest of Automattic and the web do sensemaking and public intelligence — while otherwise keeping a low profile and living relatively simply and donating any excess money beyond a typical senior software developer’s salary back into the company so it could hire more open source software developers and their support staff. Maybe my “Twirlip” work should even include better WordPress-based tools for “my” autonomous self-motivated employees to make sense of a deluge of email and help them be even more effective so they can hire more employees and bring in more revenue to expand “my” community of potential users for the JavaScript code I worked on? :-)

Of course, I could try to do that all from scratch myself instead without a friendly “takeover” or “merger” involving a turnkey operation. :-) But do I want the personal headaches of finding, hiring, and managing hundreds of self-motivated competent individuals aligned with a public purpose like making the web a better place through open source methods while still making payroll every month while also keeping investors and a billion users happy? No way! The email alone at ~1000 emailed applications per actual hire must be a nightmare. :-( Doing all those things would mean less time for design and coding and writing and so on focused on Twirlip. Best to find people who have already demonstrated a desire to do all that and have also demonstrated competence and energy in that regard including perhaps by having a pre-existing software base (maybe by a ten year “audition” period?), hire those people as a group, and then trust them to do a good-enough job at that as a community. Kind of like Uber did to the CMU’s Robotics Institute, but even bigger, since just hiring everyone is less decision making for me. :-) That would then give me more time to write grand public intelligence and sensemaking tools which all these newly hired people could support and promote alongside all the other things their pre-existing compatible software platform makes possible. But where can I find a group of people like that so I can hire them all at once to save myself time? :-) Preferably they would already have an effective management structure and leadership in place so I would not have to worry about that either. :-) And it would be nice if they already had millions of people loyally using some kind of open source software that supported decentralized publishing with an easy one-click install process which I could then build on to make it easier for people to try whatever tools I created. Oh, wait, I see such people over here… What if I were to give them ~100% of the stock in my new venture to ensure they could raise additional funding easily if needed and so the key people I delegated decision making to had sufficient voting rights to do their jobs well? Also, that way I would also not have to get distracted worrying about voting the stock (given I trust these people to do a good job of running the company, given their proven ten year track record). Why worry about ups and downs of a significant portfolio distracting me from coding? Sounds like a plan. :-)

How do NarraFirma and Twirlip connect with what you might do as a job?

So, another way forward for these ideas (both NarraFirma and Twirlip) could be to do them as part of a larger organization where they fit into the organization’s mission. That could be directly, like perhaps for a legislature or market research organization. Or that could be indirectly, like for companies that supply general hosting services, where the prospect of thousands of new customers someday might be nice, even if it might just be a small blip on the radar.

I am attracted by Automattic’s open-source emphasis for much of its work (as well as other aspects of the company). I can see the value to the WordPress community of the Automattic strategy of supporting both WordPress.com vs. WordPress.org. It would be great if Automattic could host (for pay) reliable back ends compatible with NarraFirma or Twirlip so that users who wanted convenience could pay for a service — and where other users with different concerns could instead host a backend locally or at another provider without vendor lock-in, same as with WordPress. So I feel that Automattic could be a good fit for me to pursue all sorts of projects making cool JavaScript-heavy plugins based on the general NarraFirma approach (whether involving NarraFirma specifically or not). Beyond specific plugins, I would be happy to help with creating and/or maintaining related support infrastructure, and perhaps also evangelizing the ideas — as part of a great group of people doing good work make the web a better place.

Sensemaking can be defined as “the negotiation and creation of meaning, or understanding, or the construction of a coherent account of the world.”. And as is also said there: “Sensemaking has tremendous implication for how we design and manage institutions, organizations, teams, programs, technologies.”

My hope is that eventually, the Automattic purpose of making the web a better place will include moving beyond democratizing publishing to democratizing sensemaking and other public intelligence activities. Those sorts of activities in practice are now mainly the province of governments and big organizations, who spend billions of dollars a year on them (as well as activities that spill out of them like advertising and public relations). Both my wife and I would like to see those activities become accessible by individual citizens, small groups of citizens, and organizations and communities at all sorts of scales.

Frankly, governments and big organizations often don’t do a great job at the sensemaking task. Here are examples of failures in foreign policy, economics, and likely science/medicine. And even when such big organizations do a good job of sensemaking, the results tend to be self-serving (at least to the careers of some bureaucrats). While being self-serving is understandable perhaps (even if sometimes a violation of the public trust), without an independent balancing force in a democracy like sensemaking and public intelligence done by individual and small communities that self-serving focus by large organizations is problematical (to say the least).


More about Paul D. Fernhout


tl;dr Princeton, Nabisco, the Principal Financial Group, IBM Research, Cognitive Edge, NBCUniversal; FOSS triplestores and lots of other independent FOSS projects, including a garden simulator; watching the wheels and caregiving; post-scarcity essays; open manufacturing; NarraFirma; graybeard :-)
§ Overview
§ The early years (ICs, Assembler, and BASIC)
§ First jobs after college (1985-1991; mostly in C, but some Clipper and Smalltalk)
§ Biology graduate school in Ecology and Evolution (1991 – 1993; including teaching C to biology majors)
§ The Garden Simulator and Principal Financial Group years (1993 – 1998; Smalltalk and Delphi)
§ The IBM years (1998 – 2001; mostly C++ and Java, on embedded speech, digital video, and XML standards)
§ The family-oriented decade with many small projects (2001 – 2011, mostly in Python, Jython, and Java; some JavaScript; independent writing)
§ Back to work full-time, for NBCUniversal (2011-2014, mostly in Java, working remotely)
§ On to NarraFirma (2014-present, JavaScript and TypeScript, with a little bit of PHP)
§ Next steps (maybe as an Automattic Janitorial team member making JavaScript-powered WordPress plugins to make life better at Automattic?)


Overview

So, what’s up with your career these days?

I’m a fifty-year old programmer who has somehow managed to hang in there doing programming (without transitioning to management) despite much of the computing industry being increasingly a young-person’s game.

I still like programming a lot, and while I have managed, mentored, and trained people, I’d still rather be in the front lines coding and architecting and refactoring and debugging and so on. Part of that is just being more on the introvert side of the introvert-extrovert scale. So, while I like interacting with (most) people, social interaction for introverts generally drains energy compared to more solitary pursuits, whereas extroverts generally get energy from social interactions and get drained of energy when they are alone. Of course, a good mix of such personalities is a good thing, as is said at that previous link:

“Creativity occurs in an introverted space … but that doesn’t mean we’ve cornered the market on it,” says Dembling. “Without both introverts and extroverts, things wouldn’t get done. We’ve got one person thinking it through and one person going out and slaying the dragon.”

Beyond continually learning new technologies (many of which are poorly-done rehashes of stuff from decades ago), I’ve managed to stay a programmer partly by switching from emphasizing novelty, speed, and cleverness (where younger programmers often excel) to emphasizing reliability, effectiveness, and customer service (where older programmers may have an edge). I have found that the latter attributes are valued in some settings more than the former.

For example, this Inc Magazine article says:

“People in the early two-thirds of working lives are focused on mobility, promotion, upward salary,” [Bill Coleman] explains. “People in the last third are focused on stability, something to do, reasons to be useful, and ways to interact. Work is about human interaction and community.”

The movie “Harvey” is also a touchstone here:

“Years ago my mother used to say to me, she’d say, ‘In this world, Elwood, you must be’ – she always called me Elwood – ‘In this world, Elwood, you must be oh so smart or oh so pleasant.’ Well, for years I was smart. I recommend pleasant. You may quote me.”

But all of those attributes, and more, are needed in a team that develops great software. As Scott Page writes in “The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies“, almost any good organization benefits from a diversity of individuals with different overlapping strengths and weaknesses. I feel I still have a lot to bring to the table as a programmer, even if I bring different things than I did years ago.


The early years

When did you start working with computers?

I’ve seen the growth of the personal computer industry first hand, with my initial interest driven by reading too much science fiction (especially Isaac Asimov’s books on robotics), playing with wires, switches, light bulbs and other discarded things my father brought home for me from his work at a “stop motion” company, and also watching the movie “Silent Running” (featuring the robots Huey, Duey, and Louie) at an overly-impressionable age.

My first “real” computer (other than things I built from switches and ICs) was a KIM-1 with 1K of memory that I had to learn to program in 6502 Assembler. In high school, I developed some computer-aided instructional software called PETeach in BASIC for the Commodore PET. I also won a Navy Science Award for a computer-controlled robot I developed in high school.

Beyond my father’s encouragement, I was encouraged by a high school science teacher who had a computer company on the side, as well as two other teachers who supported my early learning about computers. Thanks Jack, Joe, and David!

Where did you go to college?

I left high school early to go to college at SUNY Stony Brook, and then after three semesters there I transferred to Princeton as a sophomore (for rather dumb reasons). The summer before Princeton, I wrote a machine-language video game for the Commodore VIC-20 (where the royalties helped pay for college). I wrote that game from a home “office” (a room one of my sisters had used who had since married).

I have an bachelor’s degree in Psychology from Princeton (1985). My undergraduate advisor, George A. Miller, started WordNet just as I graduated. WordNet was tangentially related to my undergraduate work with him on triplestores and AI (and some related speculations). Working on triplestores has continued to be a hobby up to the present day (including building one into NarraFirma).

I call my triplestore system/hobby “Pointrel” for “pointers and relations”. Pointrel also means, I found out later, an engraving tool, which is also very appropriate for engraving information from the present into digital records. I asked George to add that word to WordNet, and he did. I later came to regret that request because once many web services picked up on WordNet, “pointrel” showed up in a lot of search results, and before, that the word was essentially just on my own sites (and a couple places it was used incidentally).

Of course, now with RDF and so on, triplestores are mainstream, and there are many implementations to choose from, with all sorts of great characteristics. Pointrel remains a hobby and interest, but practically speaking, it probably can’t compete with all the well-designed optimized general-purpose systems out there now, not to mention supporting SPARQL and so on.

Did you do any other computer work at college?

I’ve taken essentially no computer programming courses. I’m not saying that is a good thing, as it took me a long time to learn about hashes and “Big O Notation” and similar things on my own. However, do I have a lot of in the trenches experience. On such formative experience was for, a student work-study job, I worked at the Princeton University Computer Center (PUCC) “Clinic”.

That student job entailed helping users of the IBM mainframe with various computer challenges — mainly related to word processing with SGML and the Script program. As I wrote in “Post-Scarcity Princeton“, a person who exemplified a spirit of joy in service and who also understood work/life balance was Yvonne McCreedy, the director of the PUCC Clinic then. As one of her computer “clinicians” for a time, I learned good lessons about the joy of helping others and the habit of service which have served me well throughout my career and elsewhere through volunteerism. It just became a habit to help people struggling with some issue in the terminal rooms, whether I was “on duty” or not.

It is also true that if you really want to learn something, teach it. That job was a tremendous learning opportunity in many ways. For example, when HTML and XML came into vogue years later, they were to a large extent an application (HTML) or subset (XML) of SGML. I also started learning Unix and C on a VMUTS system set up on the IBM two-processor mainframe by another student, where 100 people might be timesharing one CPU working on their Script jobs, and the other CPU would be hogged by VMUTS running CPU-intensive C compiles.

It was also a good confidence-building exercise to be part of a knowledgeable team where you could do your part but need not know everything. People would call on the telephone or show up in person with some random question about various software packages. You might know the answer, or you might consult the documentation and find the answer, or you might pass the question on to the full-time staff who would probably know for sure. That confidence helped translate into later success consulting, teaching, and mentoring. Yvonne McCreedy and other PUCC Clinic staff members like Blair Dewey also set good examples of work/life balance and a humorous approach to life. Thanks, Yvonne. Thanks, Blair.

Those are the kinds of lessons all of a college’s students might more easily learn at, say, Berea College (a free-of-tuition work college where all the students work on campus, doing much of the labor of running it) than at a place like Princeton (where, unlike Berea, work-study on campus is somewhat a mark of shame and social divide). That time spent working at the PUCC Clinic helping other people use computers may have been the most valuable of my Princeton experiences, regardless of how much or how little it paid, and even if I may not have understood that at the time. That experience helps me see why Automattic’s policy of having all new employees do a rotation in customer service makes a lot of sense.

So how did you feel about going to Princeton?

I had wanted to quit Princeton in the middle of my three years there, or at least take a year off for a variety of reasons (including wanting to write a book about evolutionary psychology and memetics, a few ideas from which eventually became part of my PU senior thesis). Someone I knew (a friend’s roommate) had given me the notebook I first started writing about Pointrel in when I had the inspiration late at night but no empty notebook; she later transferred away from Princeton. I seemed to know an unusual percentage of people who dropped out or otherwise left the university. :-) I much later wrote a book-length essay with reflections and mixed-feelings on my various Princeton experiences.

Naturally, my parents were not very happy about the idea of me potentially not graduating. Taking a trip together with my father to Chicago to see a Robotics Exposition, he and I worked out a compromise. If I graduated, I could do something else for a year afterwards, and he would support me for living expenses for that year with money otherwise earmarked for my education. I did not realize then what a generous offer that was from a man who had done hard work all his life, first as a merchant mariner sailor and boatswain, as a bartender in San Francisco just after WWII, then as a machinist in New York, and finally as a shop foreman, toolmaker, and manufacturing engineer on Long Island. That money was left over after graduation in part because of my video game earnings helping pay for college expenses, and my getting through Princeton in three years, and my signing on for about US$10K of student loans — but the money was only there in the first place due to my father’s and my mother’s hard work and savings and frugality.

Blue Raja’s mother: [the Blue Raja’s mom walks in on him, in full costume, rummaging through her silverware drawer] Jeffrey, what are you doing?
The Blue Raja: Um, I was just, um, uh, um, I’m, uh…
[shifts to British accent]
The Blue Raja: I’m a superhero, mother.
Blue Raja’s mother: A superhero?
The Blue Raja: An effete British superhero, to be precise. I am pilfering your tableware because I hurl it. I hurl it with a deadly accuracy. The Blue Raja is my name. And yes, I know I don’t wear much blue and I speak in a British accent, but if you know your history it really does make perfect sense. The point is, your boy’s a limey fork-flinger, Mother. Hard to swallow, I know, but there ’tis. What will the bridge club think?
Blue Raja’s mother: You need more forks?

Did that experience of taking time off eventually set any patterns in your life?

I did not know then that the pattern of moving back and forth between working to other people’s specifications (especially on other people’s projects) and working to my own specifications (especially on my own projects) would recur several times throughout my career. Even were I working at Automattic, I can imagine that alternation back and forth between the two would happen — given a social interplay of defining priorities for any project through being around a lot of creative people with good ideas at various scales. Even NarraFirma itself was written to my wife’s detailed specifications as to the application user experience — even as the internal architectural specifications (including for the triplestore and panel builder and choices about Mithril and TypeScript) were mine, as were most choices about how to translate those user experience requirements into code.

Google’s “20% time” is something similar, although that is potentially a more fine-grained alternation of different projects than switching every few years. Google’s approach is more formal than what goes on in some closely-working teams. In some project teams, (a) different people have different roles and more or less autonomy within them, (b) they are all committed to the same broadly-defined self-chosen long-term goal (like a better WordPress), (c) the technical leadership shifts around day-by-day (or even minute-by-minute) as the project progresses, and (d) decisions are made in relation to different layers or components based on individual expertise or engagement. Also, given most Googlers work far more that 40 hours a week, 20% time is in practice more like 120% time, and so it is a bit like working somewhere forty hours a week and then working on your own personal projects on the weekends. I’ve done that myself all too often, but it is admittedly not good for personal, family, or community health. By the way, here are more of my comments on Google’s strengths and weaknesses in the context of post-scarcity thinking.


First jobs after college (1985-1991)

What did you do as a visitor to the CMU Robotics Labs?

So, as per the “deal”, after Princeton, I spent about a year as a (volunteer/unpaid) visitor to Hans Moravec’s and Red Whittaker’s robotics labs at Carnegie Mellon University (1985-1986), living in a small rental house I kept around 55 degrees F in the Pittsburgh winter to save money, trying to launch a career in robotics. I eventually applied to the CMU computer science graduate school (I ultimately did not get in with a Psychology degree and probably for other reasons). I gave tours to visitors to the lab (which no one else really wanted to do, and the lab staff was grateful to hand off to me), worked on a robotic finger, discussed the future of robotics and AI (from Red’s ideas for practical robots to rove through pipes to Hans’ idea of “Mind Children”), helped organize some catalogs about parts and tools, worked on my Pointrel triplestore code in C, troubleshot overheating electronics in a Kraft Undersea Robotics arm (due to a too-small power supply being used with it locally, but the board always checked out fine at the manufacturer with a different power supply), did some Unix system administration, and built a wooden model of a robot to explore the melted-down Three Mile Island plant (which helped Red secure funding to build the real thing).

While at CMU I began to become concerned about the future of robotics, both as weapons and as regards technological unemployment, but also as maybe someday becoming just effective enough to wipe out humans (a self-replicating robotic cockroach perhaps?) and then falling into disrepair or internal conflict (as ungrateful Mind Children). If I had been wiser, and more appreciative at the time of my parents’ sacrifices that made my opportunities possible, I might have used that money instead to pay off my college loans and perhaps gone back to writing video games. Maybe I might have lived frugally at my parents and enjoyed spending more time around my then-retired father (who had also suggested we could go into the CAD/CAM business together) until I could save enough to buy a house or whatever. But I was pretty dumb (or perhaps overly optimistic) back then.

What did you do as a research assistant at the Princeton robotics labs?

I then spent a year back around Princeton managing the Princeton Robotics Lab as a research assistant (1986-1987). There I did mostly C programming for Silicon Graphics 3D workstations and oversaw undergraduates using potentially dangerous industrial robots in the lab. I wrote a CAD-like application to create 3D shapes, and worked on code to plan 3D robotic paths for a robotic cutting shapes out of styrofoam, and other code in Occam (a parallel programming language) to control robots directly through Transputers which were interfaced carefully into the teach pendants. I helped write a couple of papers and grant proposals and present the work at a couple of conferences.

I really did not appreciate then what a great job working at the robotics lab was as far as learning (mastery), independence (autonomy), creativity (purpose), and social connectedness. (The items in parentheses are from Dan Pink’s work on what really motivates most people doing intellectual work.) No other regular or contracting job I’ve had (even my time at IBM Research) has equaled that robotics lab job in those ways, even if most have paid much better. And while my own independent projects have those three key features, aside from an early video game in the right place at the right time, they have never paid the bills significantly. They also generally have lacked that level of social connectedness. (PlantStudio had a supportive user community for a time, but my wife mostly managed that.)

What did you learn at that good job that applies to Automattic?

I like the idea of working with Automattic because it would once again bring all four of those things together in my life (autonomy, mastery, purpose, and social connectedness) while at the same time making it possible to pay our bills. I appreciate that combination a lot more now than when I bumbled into it with my first paying job out of college at the Princeton University robotics lab.

As E.F Schumacher wrote:

The Buddhist point of view takes the function of work to be at least threefold: to give man a chance to utilise and develop his faculties; to enable him to overcome his ego-centredness by joining with other people in a common task; and to bring forth the goods and services needed for a becoming existence. Again, the consequences that flow from this view are endless. To organise work in such a manner that it becomes meaningless, boring, stultifying, or nerve-racking for the worker would be little short of criminal; it would indicate a greater concern with goods than with people, an evil lack of compassion and a soul-destroying degree of attachment to the most primitive side of this worldly existence. Equally, to strive for leisure as an alternative to work would be considered a complete misunderstanding of one of the basic truths of human existence, namely that work and leisure are complementary parts of the same living process and cannot be separated without destroying the joy of work and the bliss of leisure.

Have you ever worked for a startup?

Next, I worked for about a year (1987-1988) at what was essentially a startup email company (Da Vinci Systems). Surprisingly, as young as I was, the company founders were younger (although one had a father who was an experienced business person).

I did C programming towards a clothing inventory management system, which was being developed as custom software for a specific client. The project had no connection with email, but it created revenue the company used to fund the email product development and launch. My work (keeping the dollars coming in) seemed to have little real value to the founders since it was just a side project to them. It was independent of their long-term hopes for the company.

I did make some suggestion about the email system design and coding standards, but that was incidental to my main work.

At one point, after having gotten some extra funding, Da Vinci Systems decided to implement a Google-ish “20% time”-like policy (in this case, roughly 10% time) as a perk. They even reimbursed a couple of hundred dollars worth of electronics I bought. I adapted a robot Petster cat to be controlled by radio signals sent via touch-tones bridging two 68HC11 microcontrollers running embedded Forth attached to touch-tone encoder and decoder chips, and programmed it to drive around parts of the office. While it was fun, I found that sort of “perk” did not change much about the nature of the rest of the work. In some ways, it even made it worse as a reminder of the sense of autonomy, mastery, and purpose I had left behind at the Princeton robotics lab job.

It didn’t help that George Miller was obviously very disappointed in my working there too. Like he said, I could have done that job even before college.

Why did you go down to North Carolina in the first place? Graduate school?

I had initially moved to North Carolina planning to go to graduate school in a PhD program in Industrial Engineering at NC State. I had been incredibly impressed by a demo I saw a staff member from there (Sam Adams) give at a robotics conference I attended through the lab, where Sam changed the Smalltalk code of 2D graphical simulations of factory robots as they ran. I had really wanted to work with Sam. And everybody (at least around academia) kept telling me I needed a PhD to really go somewhere. However, those graduate studies ended up not working out when the professor I was to work with who oversaw the related lab resigned around the first week of classes. The other faculty had different priorities.

Did you learn anything down south that applies to Automattic?

If I had been wiser, after my graduate school plans fell apart, I might have instead gone to work for SAS, which was just a couple of towns over. Someone at the Princeton Computing Center who was the local SAS expert (and originally a southerner) had recommended them as a good place to work. SAS has since become one of the top-ranked companies to work for, and is well worth studying by Automattic. Other similar computing company examples are Google today, and, historically, as ground-breaking examples, Bell Labs when UNIX was being created, Control Data when PLATO was being created (especially anything about or by William C. Norris), and Xerox PARC when Smalltalk was being created. Another example is the VM community that made IBM great by sharing source code until IBM decided to go “object code only” since “only a small percent of sites used the source code”; unfortunately for IBM, those sites included all the people who actually fixed VM or extended it for free for IBM — some IBMers were still angry about that decision decades later.

Still, SAS has a proprietary-software-based business model which is a turnoff for those for whom FOSS matters, and many SAS perks relate to working on-site. The point of such studies is not exact emulation, but to get a sense of the spirit of such places and communities at the time.

With that historical context, one can see that Automattic and the larger WordPress community may both be unique and at the same time may be one of a handful of ground-breaking computing companies that treated employees and users well and got great benefits from that. Ignoring the differences of on-site perks and proprietary software, Automattic seems to have a lot of the same spirit as SAS and those other examples as far as treating employees well, empowering them to do good work, and often fostering a larger community beyond the company that was sharing knowledge.

FOSS was not a big issue for me at that specific time when I might have explored SAS. However, a big benefit of being a contract programmer (as opposed to a regular employee) was that you could still work on and release side projects unrelated to your main job (subject to spare time). That spare time option was, and has remained, an important issue for me when otherwise working on proprietary products defined by someone else. And that was a big reason for me to discount SAS. Still, it was quite possible I could have done many useful things at SAS in the company of good people helping the world make sense of itself through statistics and other number crunching — all without the stresses of living on the edge as a sort-of entrepreneur. And who knows, maybe I might have helped SAS transition to an open source business model eventually.

Instead of listening to good advice, and having signed a year-long lease, I took the first nearby easy-to-get job that came along to keep paying the bills, having seen a flyer up on a bulletin board recruiting for a C programmer. I had no real understanding back then of how some workplaces could differ so much from others in all sorts of ways.

Did you do any innovative programming in North Carolina on robotic simulations?

I had gone to North Carolina near the beginning of the summer to get settled in before classes started, spending my small savings from the robotics lab job. During that summer, I built probably one of the first simulations of self-replicating robots in the world on a Symbolics workstation using ZetaLisp and Flavors (an object-oriented extension to Lisp). Nobody else in the lab wanted to use the Symbolics machine at the time, which broke down pretty easily and had just had started working again after repairs. That is how I ended up using it instead of the two in-demand Tektronix Smalltalk workstations that were more of a long-term prospect to look forward to. The concept was that robots would expand themselves by virtually welding on parts plucked from a sea of 2D simulated parts (including computers, grippers, movers, batteries, radars, and rocks). Then they would cut themselves in half, with the two halves intended to move away and start doing the same, like biological cells. Surprisingly, the first robot that started running unexpectedly turned cannibalistic on its own offspring. It cut itself in half as programmed, but then the original robot started cutting up the other half before its offspring could move away. I had to add a sense of “smell” to the simulated robots to prevent them from turning into cannibals (but of course, that only protected their own offspring). So I learned from first-hand experience how quickly and surprisingly the unexpected can happen with AI technology (or really, any sufficiently complex system). I learned how hard it is to think through all the emergent consequences from creating a seemingly simple set of rules (even if the risk of such an event may then seem obvious with 20/20 hindsight).

I gave a couple of talks about that work. One was at a seminar on AI and Simulation in Minneapolis. Another was at an IEEE conference on robotics in Pittsburgh. I talked about how easy it was to make robots that were destructive, and how much harder it would be to make them cooperative. I was patted on the back by a military officer (who worked for DARPA, I think) and told to “keep up the good work” at the seminar. That event ended up creating the opposite motivation, given my concerns about military robotics. In retrospect, though, I can’t be sure which aspects of my results he was praising (the destructive or the cooperative).

Did you try again to get a PhD? At Princeton?

In 1988-1989, I did a year of graduate school back at Princeton, this time in Civil Engineering and Operations Research, the same department that hosted the robotics lab. I wanted to build a mini-Google and a mini-World-Wide-Web-equivalent, especially related to organizing manufacturing knowledge. (Here are my ill-fated graduate school plans for that time.)

But, I found graduate students were treated very differently than either undergraduates or staff members, so my expectations (like for autonomy) were not in alignment with the social situation. It turns out Princeton had an informal policy not to accept its own undergraduates for its graduate programs; now I and others know why.

I could have benefited from the advice Matt Mullenweg says he would give his 20-year-old self about ambition, to go slower at first in order to go faster in the end… But I was in a hurry, and that caused a lot of friction and stumbles — even though probably at least some of what I wanted to do would have been possible with more compromises and more patience on my part.

Mr. Furious: That could work. I’m a loose cannon. I’m unpredictable. I stormed off, why can’t I storm back?
Monica: Or, you could just say you’re sorry.
Mr. Furious: Do you think there’s a really angry way I can say I’m sorry?

During that year in graduate school at Princeton I created a HyperCard stack about a “Sustainable Technology Library” called STELLA, an idea that many years later would become OSCOMAK. I’m glad to see, a couple of decades later, that such “cradle-to-cradle” ideas are now going mainstream. There is still a lot to do; the potential exists for new WordPress plugins to help the maker movement, perhaps built on new standards for encoding open source physical products and sharing the designs. It would be great to help with that, too.

How did that experience affect your feelings about academia or WordPress?

I also eventually came to realize that I did not understand much about the deeper nature of a lot of academia or the just-then-starting “big crunch“. Here is a post to the p2presearch list with some other links I’ve put together long since then about the limits and biases of academia. And another. And another. :-) An aspect of p2presearch was peer-to-peer education as opposed to (or as a complement to) more formal educational experiences.

Still, academia remains a very powerful part of our society, absorbing much of the money intended for self-reflection and self-improvement, and shaping the minds of so many people for good or ill depending on the context and the nature of the school and so on. As New York State “Teacher of the Year” John Taylor Gatto says, “schooling” is not “education”, even if some education goes on sometimes in schools.

While I can see the value in all sorts of study and apprenticeship, I can wonder if WordPress and Automattic would ever have happened if Matt Mullenweg had stayed in college. What if he had then gone on to a PhD, listening to perhaps well-meant advice that he needed one so he could “make a difference” someday and be seen as a “maverick” instead of a “crank”. Or what if he even had just gone to a more conventional high school that worked hard to create a sense of dependence?

Reading and writing and coding with WordPress is a new self-directed way to learn and grow. That experience on the web might get even better with more educationally-oriented WordPress plugins like simulations or coding tools or with improved tools for public intelligence. But that potential may still not be fully understood by many. WordPress (and the web around it) is the basis of a new form of participatory informal education (my essay) for the twenty-first century, even if formal educational mindshare and money may still be invested mostly in other directions. It would be worthwhile to help WordPress become more widely used for real education in all sorts of ways with all sorts of new plugins, with Automattic supplying related hosting services, and it would be great to help with that. In a way, Automattic itself is a bit like the university of the future.

What was your first “corporate” consulting job? Nabisco?

In 1989-1990 I worked for the Molloy Group (from my home office), mainly doing contract programming for Nabisco in Clipper and Smalltalk. Much of the work involved creating a multi-dimensional negotiation and reconciliation system for sales objectives. Nabisco had recently merged with R.J. Reynolds Tobacco Company, and while the divisions were still separate, helping a company that sold cigarettes to adults as well as sugar to kids started to bother me more and more. That was a major reason why I left that work, despite interesting challenges and innovative colleagues. I also did some smaller projects for the Molloy Group; one was helping out a producer friend of the company owner who had a sudden need for a programmer to write demonstration code that needed to be done by 9am the next morning for a commercial about a new Panasonic workstation (discussed later).

How did you respond to feelings related to that corporate job?

In 1990-1991 I spent some time doing ecologically-minded and health-minded things, in part to make up for the RJR Nabisco work. I worked as a program administrator for the New Jersey Natural Organic Farmers Association. I volunteered at the Stony-Brook Millstone Watershed association organic farm. And I was a Saturday cashier at the Princeton Whole Earth Natural Foods Store (which was one of the few non-profit retail stores at that time). That last part-time job was mostly to get out of the house and see more people. However, it had a side effect of dashing any hope I had of getting computer consulting from a few key people in the community I knew, who saw me at work there and decided I must not be much of a programmer if I was cashiering at a health food store. :-) That experience taught me an important lesson about judging a book by its cover.

I had hopes back then for creating a global organization called “Sunrise Universal Services” to help with sustainable design, with employees creating public domain knowledge about sustainable technology, living with it in sustainable communities, and consulting about that knowledge to others, all building on the STELLA idea. But not much came of it, although I learned something about organizations, equity, and rights. (Automattic and the larger WordPress community of consultants seems to reflect some of those Sunrise aspirations in a more practical and limited software context — in terms of using GPL-licensed P2/O2-powered WordPress sites in-house to support WordPress users across the globe all using GPL’d technology. That’s another reason I like Automattic)

I also helped produce a directory for New Jersey’s expanding organic marketplace using FoxBase. (My wife and I would help again with that Organic Market Directory project a couple of years later, when a new edition was wanted. That process involved designing a questionnaire, collecting survey results from hundreds of people, entering and organizing the data on a computer using a custom user interface, and producing a sophisticated report for public consumption. In a way, that directory project foreshadowed our creation of NarraFirma more than two decades later.)


Biology graduate school (1991 – 1993)

Did you give up on graduate school at that point? Or go back for more?

I then went back to graduate school and received an M.A. in Biology (concentrating in Quantitative Applied Ecology) from SUNY Stony Brook’s Ecology and Evolution department (1991-1993).

My main advisor was Lev R. Ginzburg, a Russian émigré and theoretical ecologist. Lev learned math in part through helping his father design missile guidance systems (presumably used for nuclear missiles to blow up the USA, which just goes to show how supposed enemies can become friends through common interests.) It’s funny to think that our lives in the USA depend almost totally on the reliability of 1960s and 1970s USSR technology to prevent an accidental launch. In this interconnected world, so much of our lives depend on the actions of others as part of a whole, and our actions can affect the whole in often unexpected ways.

I had known Lev because I had taken a course from him as an undergraduate. Also, I had done some work for him by writing a BASIC program to calculate statistics on oyster respiration rate data. When I had transferred to Princeton, Lev had written me a letter of recommendation. He had told me going to Princeton was a good idea as an undergrad, but I should come back to SUNY Stony Brook for graduate school, which had stuck in my mind.

I went back to SUNY Stony Brook as a graduate student to learn more about biology. I wanted to create a garden simulator that I thought of during my previous work at the Watershed association. Also, I had a passing interest in designing closed ecologies for space habitats, such as Biosphere II. (Here is a poetic essay I wrote during that time.) I was able to live at home with my parents to save money, which I think made my dad happy to have me around again.

While a graduate student at SUNY Stony Brook, I taught two courses in C programming to undergraduate biology majors. The first time, as a second semester student, I volunteered to step in when a professor developed a detached retina at the start of classes. Otherwise the class would have been cancelled and students might not have been able to graduate on time. The second time, a year later, they asked me to teach the course again because the professor had not yet returned full-time to teaching.

I also managed a library-based computing lab in the Stony Brook Biology library with about a dozen part-time undergraduate employees. I tripled the lab’s open time by placing the (otherwise stealable) manuals on reserve in the library, and by locking the rest of the problematical items and lesser-used manuals in a cabinet to which staff had the key.

During that time I also translated a numerical simulation called RAMAS/Age from the PC to the Mac for a professor’s company (Applied Biomathematics). That project included cleaning up spaghetti undocumented code. My (soon-to-be) wife was working at that company as a programmer, and she had previously been a graduate student in the same program.

How did you feel about grad school this time around?

While I had some minor negative experiences there with a couple of burned-out/obnoxious faculty (as did other students), overall I really enjoyed my time at SUNY Stony Brook as a graduate student. I learned a lot of interesting and unexpected things about ecology and evolution. I had good reviews from the students I taught, and very few other graduate students were teaching their own courses (maybe just one other in the department). The university even gave me a faculty parking permit, which was very convenient for a commuter student. I met lots of interesting people, including Larry Slobodkin, who had founded the department and sort of took me under his wing, letting me play around in his lab, taking me rowing at Swan Pond to collect Hydra, inviting me to his home, and so on. I might well have stayed in the program if I hadn’t met my wife (who had left the program already).


The Garden Simulator and Principal Financial Group years (1993 – 1998)

So, what did you and your wife on work on together?

Next, my wife and I worked for a year on implementing my (now our) garden simulator (1993-1994). Despite all the news reports around that time about the need for non-violent video games, games that women would want to play, and the need for more educational software, we could never figure out how to convert any of that into funding for my (now our) garden simulator project. As our credit neared its end, I needed to get a real job, even with the garden simulator still unfinished.

So, what was your second job for a big corporation? Insurance?

So we moved to Iowa, where I was a contractor doing Smalltalk programming and mentoring, for about two years total, for the Principal Financial Group (PFG, 1994-1997), with a one-year gap in between. I developed GUI front ends in Smalltalk, the last of which (called TeleApp) supported hundreds of thousands of insurance application phone interviews.

During the first year, I informally mentored other Smalltalk developers. I co-developed a new in-house GUI framework to standardize GUI construction using the “Mediator” design pattern. That standard made it a lot easier to understand and maintain code, compared to having to understand several different styles of coding GUIs.

At one point, I refactored a whole bunch of code (with prior permission) when most of the rest of the team went for a one week training course. That involved removing the C-style get/set prefixes on thousands of methods and making them use standard Smalltalk conventions (such as to use “x” and “x:” instead of “getX” and “setX:”). The team’s prior trainers had a C++ background and taught bad habits from a Smalltalk perspective. I created a tool to help with that refactoring. I also wrote code that generated reports in advance to determine the few places that the new methods would conflict with existing methods to determine workarounds. It was a big undertaking, but the results made the code look a lot better from a Smalltalk perspective.

I had to wear a suit when I went in, but somehow I did not really mind that very much.

During the second year of the work for PFG, I worked from my home office, only donning one of my suits when I went in for occasional meetings every few weeks. At some point that second year, the Principal went “casual”, which was a huge thing for the office.

My wife continued to work on the garden simulator, and I worked with her full-time our second year in Iowa, before doing another contract for a year with PFG.

In retrospect, we should have gotten Cynthia working at the Principal (they even asked if she was available), at least the first year. The first year in Iowa was very tough on my wife, as she was pretty lonely being in a strange place with no local friends (until we moved from downtown Des Moines to Ames). But we were (foolishly) afraid of being “scooped” if we stopped work entirely on the software, or of just never getting back to the project.

It did not help that she spent much of that first year trying to untangle spaghetti FORTRAN code written by the USDA ARS — even though she succeeded. Wow, were the USDA ARS researchers surprised when we eventually visited them down in Blackland, Texas, presenting them with the rewritten model code — especially surprising to one of them because a “woman” had done most of the rewriting of that code which only one guy in the whole place seemed to really understand.

What did you do in Iowa when not writing insurance applications?

My wife and I have worked on several independent software projects since then, as “Kurtz-Fernhout Software“.

We used our profits from the PFG work to finish our free Garden Simulator software.

Garden Simulator screenshot showing all four main windows

Garden Simulator screenshot showing the four main windows

We also created a 3D botanical plant designer tool called PlantStudio

ps

…and a voice-activated choose-your-own-adventure designer called StoryHarp.

sh

All three titles were released around 1997-1998. They used the same common core classes (which I designed, roughly based on MacApp), though with initial design work by me stretching back all the way to around 1990 on the garden simulator. The software was written mostly in Delphi, with some of the earlier work done in Smalltalk and C++.

Of the three, PlantStudio was the biggest success. PlantStudio and StoryHarp were shareware for a time, and then made open source. Unfortunately, by the time PlantStudio became a minor success, we were forced to be doing other things to bring in money, otherwise we might have been able to kindle that spark into a flame.

How did you end up going free and open source?

We had originally intended all three titles to be shareware, with the goal of using the money to write even more educational software, but we ran out of money just as we finished them and couldn’t afford to keep supporting them full-time. Also, my wife had put so much time into translating spaghetti public domain FORTRAN code from the USDA Agricultural Resource Service into beautiful readable Delphi code that formed the core of the Garden Simulator that we just could not bear the thought of not making those improved scientific models available for further review and improvement. We had many long discussions about whether to go open source or not.

We had a bit of hope for moving the Garden Simulator forward with a 1997 NSF grant pre-proposal on informal science education for a community-driven Garden Simulator 2.0, but the response we got back was discouraging. The NSF did not yet get “open source” or see the benefit of free software as a way to spend public money (although we also could have done a better job of selling the idea of open source software funded by public dollars).

Source code for the PlantStudio and StoryHarp projects can be found on my pdfernhout GitHub site. The source for the Garden Simulator is on our kurtz-fernhout.com website.

We were especially proud of what our PlantStudio users had to say about the software. One example:

“The interface is good, you’ve obviously put a lot of time and thought into it (even the pop-up help screens are good), the program does what it claims very well, and the bottom line for me is that it’s immediately fun to play in and gives me all kinds of new ideas and possibilities (it’s that “oh, wow, this is neat– hey, I could try this– I wonder what happens when I do this– you could do a bunch of these and combine them and then….” factor, and two hours later you’re still messing around and getting ideas — instead of “now what, so what, this is really awkward, this doesn’t go anywhere”.”

Much of that credit has to go to my wife, who is a much better GUI designer than me.

What have you done with that garden simulator and other code since?

Delph was a proprietary system and I wanted to move the code into something non-proprietary. (This was before Lazarus, a Delphi clone, really got good.)

I made some effort to port all three pieces of software to Java and Python by writing a tool to convert Delphi code to those languages, but there was still a lot of hand-coding required to make them work. I only got as far as getting StoryHarp to work under Java, and part of PlantStudio to work under Python for the OLPC Sugar environment, before other things took up my attention.

Someday I’d like to rewrite all three as JavaScript web applications (or help others do that). I’ve interacted with a few people over the years interested in the source code, but no one has had a sustained interest in it.

While I believe in open source (as opposed to artificial scarcity), the fact is, applications are often complex beasts that have big learning curves. Some of the learning is about the software, but often a lot of the learning is about the problem domain. For example, it took us years of learning about gardening, soil science, climate modelling, and so on to know something about what the USDA ARS models did. You can’t just pick that up in a couple days the same as, say, reading through the Backbone source code. Complex applications as interwoven wholes usually need a lot more love and care than smaller bits of code like libraries, or even than larger libraries that are easily subdividable into mostly independent units such as Apache Commons Math. I think that is maybe part of why there are so many open source libraries but a lot less open source applications.

For many years, I was always saddened when I’d come across some tax-funded or grant funded software application (generally developed at a university) only to find it was either for sale, or, if freely downloadable, it did not come with source code (so that I could learn from it or perhaps make it even better). I felt after all the time we had put in on the garden simulator, I had earned the right to complain about that situation of tax funds going to create proprietary software . :-)

Why did you leave Iowa?

We loved our time in Iowa, apart from being far from family. For much of that time we lived in a beautiful four-bedroom farmhouse in Ames, Iowa. The rent was relatively cheap, and we could walk to the Iowa State campus. We lived across the street from a 150 acre arboretum where we could walk our dogs. People were friendly. A friend of my wife’s from graduate school moved out to the same university town our second year in Iowa, and they took art classes together.

My plan to pay back our remaining debts had been for me to take another high-paying Smalltalk job, either in Iowa or nearby in Kansas City, where a contracting friend said Sprint was hiring Smalltalk programmers at good rates. However, around that time my father (nearly eighty then) suffered a small stroke or seizure while on a trip to Europe to buy a Volvo direct from the factory. It probably had only added to the stress of the trip that I had strongly discouraged him from going on it. Luckily the event happened when he was at a relative’s house, and they could help care for him and my mother (who had diabetes and increasing dementia). When he came back to the USA, his doctor would not let him drive for several months, which made life very hard on Long Island. Not knowing if he would ever be able to drive again, we made plans to move back to New York to help out and so I could spend more time with my father.


The IBM years (1998 – 2001)

Why did you work on-again-off-again at IBM?

After those projects, to pay the bills (and pay back the credit cards we had run up creating our garden simulator), I was a contractor with IBM in New York for about two and a half years total, mostly in C++ and Java, working on three different projects (1998-2001). IBM had a policy that contractors could only work up to eighteen months at a time, with a six month break in between, to keep them from being ruled as employees.

How did you feel about moving back East?

Westchester County in New York (near IBM) was an expensive place to live, as was New York in general. And IBM, as a name brand, paid much less than I had been getting in Iowa.

We felt we had to drop our health insurance as New York rates were more than double what we were paying in Iowa in order to pay back debt faster, and then rates just went up from there. Ironically, my parents, who we had returned to help care for, were covered by four different overlapping health insurance plans. Cynthia and I were not to have health insurance coverage again for another fifteen years or so. Lack of health insurance was an ever present stressful worry in the background of almost all the rest of our time in New York trying to remain independent free software and free content developers as much as possible — a worry that free software developers in, say, Europe don’t have. Thankfully, the gamble paid off and we never had a major unexpected health issue.

In New York, we also ended up paying more than double our rent in Iowa. We had moved from a joyful Iowa college town renting a beautiful large well-located well-maintained house to instead renting a smaller rundown house in the slums of Chappaqua walkable to the train station (my wife had also gotten a contracting job, working on Wall Street). That house was one of the few rentals we could find in the larger area on short notice that would take two good-sized dogs (in part because it was so rundown).

Our household goods were delivered in an ABF freight trailer with several pieces of furniture damaged. Unlike my father, I did not know how to pack a vehicle well, and I had also just not understood some good advice by a friend in Iowa, a janitor who had also done moving, with my not realizing how much stuff would bounce around against each other. As I was moving boxes from the trailer to the house late at night, two girl gangs in new-looking cars (perhaps from the next town over) started threatening each other with harsh words about one group driving down the street of the other group, accusations about one girl being pregnant from a teacher, and then a threat about a knife — all while I unloaded our stuff a dozen yards away. I wondered if maybe they wanted an audience? I asked Cynthia to call the police, mostly so they would not escalate things further to physical violence — although both cars ended up driving off on their own as a police cruiser came by (perhaps someone else called first). Nothing like that ever happened again, but I was rarely out that late again there.

People trudging home from long train rides from New York City would glare at you or tell you to mind your own business if you said hello. My wife’s dental hygienist told us how much she disliked the schools, which were rated among the top ten in the country but which discouraged any sort of non-academic aspirations (to keep their college acceptance rates up). Sometimes over-stressed kids would shout random insults to passers-by in the streets. It should have been a big tip-off when the realtor who rented us the house joked that you knew Chappaqua was a top school district because the teen suicide rate was so high. :-(

Chappaqua was an unhappy place to live for us for those reasons (and more), especially after a joyful sense of community when living in Ames, Iowa.

From a The Culture of Affluence: Psychological Costs of Material Wealth, a study by Sunita Luthar that I only discovered many years later and reflected our own experiences of Chappaqua:

Children of affluence are generally presumed to be at low risk. However, recent studies have suggested problems in several domains — notably, substance use, anxiety, and depression — and 2 sets of potential causes: pressures to achieve and isolation from parents. Recognizing the limited awareness of these issues, the objectives in this paper are to collate evidence on the nature of problems among the wealthy and their likely causes. The first half of the paper is focused on disturbances among affluent children and the second half is focused on characteristics of their families and neighborhoods. Widespread negative sentiments toward the rich are then discussed, and the paper concludes with suggestions for future work with families at the upper end of the socioeconomic spectrum.

We did not have a kid then for whom to be directly concerned about the terrible oppressive schools and such — just two dogs (who felt to us at the time like they were our kids) and several “mind children” software products. But the spillout from the messed up “best in the country” schools was still unpleasant to be around. (That experience was one reason we ended up homeschooling/unschooling years later somewhere else without huge school taxes…)

Things got even worse in some ways after the Clintons moved into Chappaqua some months after we did. Although, had we bought instead of rented, the sudden property appreciation would have funded a lot of free software — not that we expected we could have gotten a mortgage with all the debt still piled up from focusing on our independent projects.

But the good news was that I got to see my father every other weekend or so, after a two-hour drive through NYC traffic to get to my parent’s house.

And we were also closer to Cynthia’s family as well.

Were there any other aspects to your health insurance choice or its consequences?

There was another aspect to that choice about health insurance which ultimately lead to a proposal for improving health sensemaking on the web.

I am all for Medicare/Medicaid for everyone funded by progressive taxation (especially in the context of a holistic wellness-promoting health care system that emphasizes prevention, education, conflict-of-interest-free advice, and good nutrition, such as Dr. Andrew Weil writes about or others like Dr. Joel Fuhrman and many others would suggest). However, it is basic psychology in a mind-body sense that, while buying health insurance (really, sick care insurance) is sensible for financial reasons, choosing to buy sick care insurance is bad psychology for mind-body reasons because the only value in it is when you get sick. While sick care insurance does have peace-of-mind value, it also creates a big of negative psychology towards sickness in that personal choice.

Also, the vast majority of sick care costs go to artificially prolong the suffering of people at the end of their lives (“frailspan”) or to help those who have made poor lifestyle choices for a long time keep making bad choices as well as ignore the social context that puts people in so much distress they make poor choices (whether illegal drug addiction or too many refined carbohydrates). As Dr. Fuhrman says, a prescription for many drugs such as blood pressure medication is like a “permission slip” for continued bad behavior.

I am all for helping people live healthy long lives as well as to care for those with health problems — but my choosing voluntarily as an individual to underwrite such a sick care system seemed morally problematical to me as a personal choice. (Not that I have such a choice legally anymore.) Also, if you were to ask most people, “Would you rather live one or two years longer in suffering at the end of your life, or would you rather see your children grow up with lots of care whenever they got injured physically or mentally, able to take a year off from work to spend taking care of your newly born grandchildren, and having the free time to bring your grandkids to visit you?”, I’d suggest, given the actual choice, most people would choose the latter option. But our “health” care system is not set up that way and does not provide that choice. I know of course that this is a complex moral landscape, more complex than I outline here. And, actually, I feel we have enough resources if we want both options, but that’s another story.

When the cost of sick care insurance was small as in Iowa where costs were rated based on our age, it was an issue that was easy to ignore. When the costs we much larger in New York which does not allow age rating on premiums, that issue was brought more to mind.

Another consideration was that the money otherwise going to the sick care industry “just in case we got sick down the road” could instead go to things like having dogs for stress reduction and exercise in the sunlight, or buying more health-promoting produce and other organic food, and making other such choices like YMCA memberships or buying a Vitamix blender to reduce the likelihood we would get sick. People sometimes would comment on the amount of produce we buy at the store (which admittedly has dropped off recently under stress and a retreat to refined carbohydrates). I had a related conversation once with a grocery store clerk doing bagging, an older guy, who said he could not afford to buy much produce, but who spoke about probably needing a heart operation. He was glad to have insurance coverage for it if needed (perhaps a US$50,000 cost). Of course, his insurance would not have subsidized his purchase of vegetables at a much lower cost — which could have potentially reversed his heart disease non-surgically. Is that the sort of financial system I want to underwrite as an individual choice?

Also, the one time we wanted to significantly use our sick care insurance in Iowa to cover physical therapy for a bicycling-related knee injury, it involved many hours of phone calls and so on and bewildering paperwork both before and after to get partially reimbursed (including after claims were initially denied for lack of pre-approval even despite actually being pre-approved as an “unnecessary” precaution according to the insurance representative). That was a very minor experience of the much greater horror stories in Michael Moore’s movie about health insurance failure called “Sicko”. At least when the money is paying off our credit cards, we knew where it was and could in theory use it to pay for medical expenses (up to a point).

Given all those sorts issues, as well about US$110K in credit card debt when we moved back to New York (eventually repaid on time with much interest over the next decade), it was hard for me to want to buy into such a sick care system voluntarily until legally forced to.

That fifteen years without sick care insurance was a great spur to pursue an interest in health sensemaking as an attempt to move beyond the worst aspects of “The Google Health Plan“. :-) That is learning and sensemaking I’ve tried to help everyone with, from grocery checkout clerks, to Slashdotters, to billionaires, to the then Governor of New York State, and even in an open letter to the President’ wife. :-) Perhaps annoying so, admittedly. And I’m not saying any of them listened or that what I did was effective. I just felt I had to try. As I still do even now, with some basic health ideas woven into this document.

Also, as my wife’s sister is a medical doctor, we could also get some occasional free advice to double-check things. That is an option not available to everyone without health insurance. She also told me about Dr. Andrew Weil’s writings which I have learned a lot from. So, thanks sister-in-law!

In the last few years, I learned about David H. Murdock, a ninety-plus year old billionaire (from Dole Foods) interested in health issues and plant based diets. He endowed a research institute in North Carolina related to that. So, between Google’s founders and him and others, I can hope we get a much better medical system eventually. For a billionaire, there are probably few better investments than some money spent on improving health care for everyone. However, Google’s approach is of course centralized and machine-intelligence intensive, and the North Carolina Research Campus approach seems a bit academic and also biotech focused. IBM’s Watson is also planned to be used for medical assistance, but it of course will likely only regurgitate mainstream medial advice from a literature which has been systematically biased by huge conflicts-of-interest situations and other problems. So, I still feel there is room for individual contributions in this area, like better software tools for health sensemaking and more comprehensive content that integrates the best from the past with the results of more modern experiments.

What did you work on at IBM Research? Speech recognition systems?

At IBM Research, in the Speech Group, my first and longest contract with IBM, I helped to implement the IBM Personal Speech Assistant (PSA). This was a special speech processing hardware that ran alongside a Palm Pilot (a forerunner of Siri and such). I was co-inventor on a patent for this (even as I feel software patents are a bad idea). About a decade later, an Apple recruiter contacted me out of the blue and I guessed (correctly it turns out) that Apple was working on something like Siri and had probably seen my name on that patent (although I did not pursue the job for various reasons back then).

Most of that speech work was in C++, with a little bit in Python and Squeak. The C++ was used both with Microsoft Windows on the desktop (used for remote transcription using IBM’s desktop ViaVoice software) and for VxWorks on custom embedded hardware we were working on. I also created embedded Squeak then, on my own time, but with the hope to use it in a project at IBM. The IBM Chairman at the time, Lou Gerstner, asked for a PSA for his office, which was a huge political win for my supervisor (“Master Inventor” Liam Comerford), even if the technology itself did not become a product. The PSA served at the very least as a good demo for IBM’s embedded speech technology (also useful in cars and such). It also resulted in various patents.

During that time I also worked on an independent project in the Speech Group. I created a speech-operated display wall using nine retired ThinkPads otherwise destined for “the crusher”. It was a proof-of-concept demo for making a display wall with hundreds of old laptops. A couple of regular IBM employees I was friends with (one who invented VGA and another who had been a fashion photographer in NYC for a time) helped me get the equipment for the lab. Obtaining nine laptop security cables was the hardest part of it, to comply with IBM policies on laptop use. Another contractor helped with setting up Linux for the server, and another helped me scrounge up a wooden stand for the laptops. The result looked a bit like a “Jeopardy!” wall, and you could ask for different things to be put on different screens.

IBM’s old-equipment-recycling division was interested in the idea, because no one wants to see beautiful ThinkPad hardware destroyed just because it’s a bit out of date CPU-wise. I had suggested that perhaps ThinkPads could even be slightly redesigned to make later use in display walls much easier. But as my eighteen months was about up, the idea got dropped (as far as I know). Another speech researcher took over the display wall when I left, though. I spoke with the head of IBM’s legal department about maybe using something like a display wall to help with patent research and writing, but that idea did not go anywhere either. I did that project when my supervisor went on vacation for two weeks near the end of my time there. I didn’t ask him about going ahead with it, so he couldn’t say “no”, and wow was he surprised (understatement) when he came back. :-) He’ll still give me a good recommendation though. :-) And people did say it was better to ask for forgiveness than permission. And Liam had earlier said, in the good old days at IBM Research, many years earlier, an interesting idea like the speech-operated display wall project would have just been done as a matter of course once suggested.

[Mr. Furious tries to balance a hammer on his head]
Mr. Furious: Why am I doing this, again?
The Sphinx: When you can balance a tack hammer on your head, you will head off your foes with a balanced attack.
Mr. Furious: And why am I wearing the watermelon on my feet?
The Sphinx: [looks at the watermelon on Mr. Furious’ feet] I don’t remember telling you to do that.

While I was at the Speech Group I also got IBM Legal to officially approve Python for use at IBM Research, which included IBM Legal writing a letter asking Guido von Rossum if he really did write the software. But those were the early days of “open source”. I also got Squeak approved to use at IBM Research, but only in a very narrow context given the different (more viral seeming) license it had then.

The Speech Group did talk about my becoming a regular employee and took steps to prepare an offer, but I was still not ready to leave behind my independent aspirations to develop more educational software and information management tools. At IBM, you had to sign over anything you ever did or thought about (at work and outside of work), although some people there joked that many IBMers “had really smart spouses”. While I could have accepted that, as it was explained to me, a “success” meant that whatever you developed would be given to another IBM division. You would have a few weeks to transition the product “over the wall” to that division, after which you would never be allowed to work on it again. And of course a failure was not good for your career. Also, I was told that IBM Research focused on the “Bs” (billions) and not the “Ms” (millions), meaning you could have terrific useful ideas that would get shelved because they were not potentially profitable enough. IBM Research was a fun place to work back then, but if you really cared about what you were doing, it could be painful. IBM back then was also still just starting to get “open source”. Today’s IBM Research might be a very different situation, as with Eclipse, Jikes, and similar projects. So, when my eighteen months were up, I went back to working on our own software.

What did you work on next at IBM? Digital video?

Still needing to pay the bills, though, about six months later I started work at the IBM Internet Media Division. I worked on URL injection into MPEG-2 video streams, which supported related e-commerce web content served using WebSphere. Going beyond the original job concept I was hired for, I got the whole system supplied by research to work; it involved a web server, a QAM modulator, a set-top box, and a PC with the special “MediaPump” card I was writing code for. I also provided some pre-sales support for that system to IBMers outside of research trying to sell the system to cable companies. I also made suggestions about how IBM might do well by selling hard disks optimized for storing digital video at home.

After that, what did you do with XML standards?

After another break, back at IBM Research again later, in the XML standards group, I helped implement an XSL-FO parser and 2D printing tool in Java as part of the XSL-FO standards process. That work was much like creating a web browser rendering engine like Gecko or WebKit, except that it was focused on creating printable pages. To support regression testing, I created a tool in Python, and then Jython, which would run through our application about 1000 test cases we had developed or gotten from other people, save the results as images, compare the images with those from previous runs to see if the bitmaps had changed at all, and produce a report on the changes as an HTML page, where you could then click to see what changed between runs. That tool helped us to make progress on the code with more confidence (somewhat like unit tests). That software became part of IBM’s AS/400 printing offerings and was also on IBM AlphaWorks for a time.

What was your wife doing around then?

In between those projects I worked on my own projects, like the Pointrel system and StoryHarp. My wife also started working at IBM Research around then, in work related to stories in organizations. She had previously been commuting into New York City to work on Wall Street doing technical writing to help pay our debts, but the two hours or so commuting each way was very wearing on her. She would call me from work sometimes at 5pm or so, so sad that she was facing a long commute back. I had seen an opening for an IBM Research job (as a “supplemental”, so hourly with no benefits or job security) and passed it on to her. She has continued in that line of work since then — including designing open source software that includes NarraCat, Rakontu, and NarraFirma.

XML was big back then, especially at IBM. With my knowledge of XML standards and XML Schemas from the XSL-FO work, I helped my wife in a small way with the practicalities of realizing her idea of creating StoryML, an XML Scheme to mark up stories with metadata. StoryML got mentioned to someone at DARPA, which eventually led to further career opportunities for her.

Did anything else happen around then to influence your feelings about open source?

Around that time, Jim and Janet Baker lost hundreds of millions of dollars through apparent fraud and deception when they sold their Dragon Systems speech recognition company to another company. They were paid mostly in stock in the other company, and the stock was overvalued through questionable means. The Bakers had both studied with my undergraduate adviser at Princeton, who was an expert in the psychology of language. They had also both worked with the IBM Research Speech Group before I was there (although people disagree about which way most of the knowledge about speech was flowing). Losing most of your life’s savings, which resulted from years of hard work and intelligent risk-taking, has got to hurt no matter how it happens or how much is lost (even if you are left as “just” a millionaire). But what really hurt them, George Miller told me, was losing control of the source code for the “baby” they had worked on together for so long.

That’s a big reason I like the idea of working for an organization where most of the work is open source — to be able to at least in theory stay connected to “mind children“. Companies can change — as the history of ArsDigita and Philip Greenspun accepting Venture Capital shows as a cautionary tale (although that is from the days before open source was more common as a business model, and even with the company going under eventually from management conflicts, the OpenACS code remained available). Whatever happens to Automattic in the future, Automatticians can sleep well knowing they will always be able to stay connected with the GPL’d WordPress codebase. And for a creative person, if you deeply believe in free and open source software as a public benefit (as I do), then a FOSS approach translates into more confidence that the codebase is worth pouring your heart and soul into (well, within the limits of family and community obligations and a healthy long-term balance in life). There is little reason to hold back good ideas from a FOSS-based business like Automattic just because you might want to independently work on them later, like I saw happen at IBM Research. There is little reason to fear that implementing a good idea for a FOSS-based business makes the idea the “proprietary” property of someone else who can exclude you from ever working on what you created again.

By the way, I can speak relatively freely about my time at IBM now only because the ten-year confidentiality clause in my contract has expired. That’s typically another personal cost of developing proprietary software, not being able to talk about what you do. At least we were not in the position of some relatives of Cynthia, where a husband and wife worked for competing research companies and so could never discuss work with each other.

Towards the end of that time, and during the beginning of the next chapter in my life, I participated in discussions and some coding related to Doug Engelbart’s Bootstrap Institute and his Unfinished Revolution II (UnRevII) Colloquium run by Stanford. I took part in that colloquium remotely, getting a completion certificate from Stanford’s continuing professional education program. I also spoke with Doug once on the phone to discuss open source licensing options for new software related to the Bootstrap Institute.


The family-oriented decade with many small projects (2001 – 2011)

So, how do you explain a resume gap of a decade without full-time employment for others?

After working with IBM, I was frankly a bit burned out on outsider-status contract programming for really big companies (even as I enjoyed interacting and discussing ideas with colleagues and liked the R&D environment in general). For example, I met the brilliant Sam Adams again at lunch one time at IBM Research (Sam having joined IBM eventually a few years before I contracted there). Cheered on by our unexpected meeting, we started having a fascinating conversation — until he realized that I was wearing a contractor badge and he had assumed I was a “regular” employee at first, and that he (in sorrow) could not tell me much more about what he was working on (it being “IBM Confidential”). There were also planning strategy meetings I might have liked to take part in, but which I could not as a contractor. But having passed up a full-time job in the Speech Group, I can’t blame IBM for “outsider status”. I also wanted to get back to working more on our own projects, as we had made progress paying our debts.

In October of 2001, my father died suddenly at age 85 from a heart attack (about a month after the 9/11/2001 attacks). I needed to step in to help take care of my mother, who had dementia and diabetes. My father had been caring for her, including administering three insulin shots and four finger sticks for blood sugar monitoring a day.

How do your feelings about your father’s death relate to free and open source software?

When someone you are close to dies, you have all kinds of feelings about that, both immediately and long term. Some make sense logically, some don’t. In my case, I had just been finishing a first Java version of my previously Smalltalk and Python based Pointrel triplestore system, in large part because people in Doug Engelbart’s UnRevII colloquium were more interested in Java than other options (and Sun was providing some support to Doug’s effort). I literally saved the last change for the port in the last example application as I reached to answer the phone call that would give me the news of my father’s death. I had not visited my father as planned the weekend before, mainly because of our two big dogs had pulled on their leads unexpectedly while I was going down some outdoor stairs which had strained my back and would have made the trip harder, but also because I had wanted to finish that software (a version which in sorrow I just left sit around unreleased for years). I so regretted not seeing my father one last time that very last weekend when I had the chance — even though I had seen him dozens of weekends in the preceding years since we moved back to New York (for about eighty or so weekends in total when Cynthia and I counted them up once to try to make me feel better about missing that last visit, but that count still didn’t help much).

I later wrote a related essay called “The High Price of Free Software”, about the personal and family social costs of time spent coding on open source software projects. It was in a way an echo of an essay I had read in high school about CS students at Stanford “losing elements of their humanity” with too much time spent at glowing screens — something you think won’t happen to you, until it does. Yvonne McCreedy at the PUCC and others I had known including at IBM Research had presented a model of work/life balance; it was not their fault if I had not followed their good examples as well as I should have for whatever reasons. Politically, something like a basic income might help many people address that balance of working on projects of unfunded personal interest that may also help the world while still having time for family and other obligations, but we don’t have a basic income yet.

Is there anything from that experience and thoughts on work/life balance that relates to open source and Automattic?

That experience is part of why I would be appreciative of working at Automattic where I could feel satisfied putting in a good day developing open source software as part of a good team and then not feel I need to do that on weekends and evenings too (as when either developing proprietary software for a day job or trying to get things done as an independent on a shoestring with our bank balance steadily ticking down as with NarraFirma). As the npmjs community puts it:

npm is not a typical product, and we are not a typical early-stage “work hard/play hard” startup. We are responsible adults with diverse backgrounds and interests, who take our careers and our lives seriously. We believe that the best way to iterate towards success is by taking care of ourselves, our families, our users, and one another. We aim for a sustainable approach to work and life, because that is the best way to maximize long-term speed, while retaining clarity of vision. Compassion is our strategy.

As a favorite author, James P. Hogan said about moving to writing full-time, when people asked him if he would put out more books, his answer is that he would put out the same amount of books, but he was now going to have more time for family and friends.

As Jenny Santi and many others like Richard N. Bolles suggest, better work/life/play/learning balance can help avoid burnout over the long term.

One related thing to consider regarding work/life/play/learning balance is Automattic’s vacation policy. Normally, what could there be to complain about with “unlimited vacation days”? :-) Let’s ignore the fact that as with web server hosting “unlimited” is never really “unlimited” (which is one reason we host somewhere with guaranteed quality of service and allotted bandwidth and disk space and so on). In practice, Automattic knows the kind of devoted people who work there are not going to take much vacation by choice, so even if a few do for whatever reason, it’s not a big deal for the entire company. The main problem is going to be Automatticians not taking *enough* vacation. It’s problematical to ask employees to self-regulate a minimum vacation because it sets up an implicit conflict of their loyalty to the organization versus other needs. Also, Automatticians are typically going to see work as so engaging (including because they are also learning, having fun, and making a difference in so many people’s lives each and every day) that it may just be hard to step away from laptop or smartphone connected via VPN to the rest of the Automattic team and the larger WordPress community. But too much of a good thing can have some non-obvious negative consequences. (I’m sure the same might be said about the length of this essay.)

So, as has been suggested elsewhere (I think I probably first saw the idea in an Amazon reviewer’s comment on Scott Berkun’s book), it might make sense to enforce a *minimum* vacation policy at Automattic. Here is one way to do that. Four calendar weeks a year an Automattician’s VPN access could be restricted and they would be asked to take that time off and go do something else — ideally not involving the computer. Plant a garden, volunteer in a soup kitchen, travel to relatives, train a dog, learn to cook better, just sit around moping around being disconnected — whatever they want to do. If they want to take more time off than those four (or whatever) weeks, then of course there is still the “unlimited” policy. These weeks off would be in addition to sick days taken and so on. The specific weeks off would at an employee’s choosing, but people have to choose the minimum number of weeks or accept some default choice of weeks. Probably VPN access would have to be restricted to help people resist the temptation to work on their vacation. Otherwise, no matter where Automatticians are, you know they will probably pull out a laptop or smartphone and just “check in for a moment”, and spoil the gathering psychological momentum of the vacation.

Here are four benefits of such a minimum vacation requirement. First, vacation is good for the mental health of the employee and may also spark new ideas that may require some downtime and indirect reflection. Vacation helps ensure long-term employee productivity for those and other reasons. Second, sadly, it has been said, “The graveyards are full of indispensable people.” Anyone can get hit by a bus or have some other unexpected tragedy befall them. Vacations are one way an organization practices for when someone might not be around. That practice includes personal growth for people who need to step into big shoes for a short time. Such short “trials” filling in for others can also help everyone decide whether they’d like to do such tasks more often or not. Such experiences generally help spread knowledge around the organization. That spread of knowledge can be beneficial in other indirect ways, as people come to appreciate the work others do by understanding the complexity of it a bit better. Granted with P2/o2 blogs, much knowledge is already spread around throughout Automattic, but there is often a difference between tacit knowledge (information often assumed, often unspoken, and often broadly contextual) and explicit knowledge (information which is formally written down or easily verbalized). I’d suggest even Matt Mullenweg could be in on such a vacation policy for those reasons (may he live long and prosper despite such a precaution, of course). Third, vacation also builds in an emergency buffer (as one way of several other possibilities). If some really serious unexpected crisis hits, then planned vacations could be moved around. Having to move vacations for a crisis is essentially an admission of a systemic planning failure somewhere in the organization, which should then be addressed in the future, but it is still good to have potential spare capacity that could be brought immediately online. If about one-tenth of the staff is always on vacation, then you can in a worst case call everybody in from vacation and surge the entire organization’s output by about 10% for a short time, cranking output up from the usual 10 to about 11. Fourth, vacation is good for society in general, including because vacation provides time to reaffirm face-to-face social ties and to stay involved in local subsistence activities, the informal gift economy, and various civic duties like public participation in government planning that all together make a society stronger and more resilient.

Non-US countries do things differently so thing might need to be adjusted globally. For example Germany federally mandates six weeks of vacation a year, yet their productivity is higher than the USA’s.

Would this suggestion work exactly as proposed? Maybe not. And maybe disconnecting VPN access four weeks out of the year may be too drastic or cause other worries. Maybe people on vacation would need some way to get daily news not to worry? And who want to go on vacation to come back to a huge pile of email? The current Automattic vacation policy may well be working better in its current cultural context than I might guess it would. So, just something to think about.

How do you feel about Automattic meetups as described by Scott Berkun and how they reflect work/life balance?

I would certainly be willing to put in long work weeks now and then at meetups if that is the Automattic culture. It might be fun, and even if it wasn’t, working with Automattic has so many other advantages that outweigh occasional travel issues.

Nonetheless, I still find this Amazon review on Scott Berkun’s book insightful:

Automattic has a completely distributed team around the world, and a scarcely used San Francisco office. The company pays for relatively frequent meetups among employees from all different countries. These meetups in various attractive locations (Athens, New York, Lisbon, etc.) sound compelling, but it is clear that when people show up to these events, they have close to zero time for glamor or leisurely exploration of their surroundings. It is a matter of working, with occasional “biology breaks”, for very long hours, while they have the chance to be in the same room together. Nights spent at classic arcade spots or pubs with a good selection of games might not sound too thrilling to some. This part of the culture could definitely scare away women employees. [my emphasis]

At least for me, while I like new experiences, and I even enjoy pair programming, I’m generally not going to be coding at my best for some tasks in strange surroundings with lots of distractions. That can be true even though I have done that in the past with success. That included once in the late 1980s coding up demonstration software for a commercial that was set to film the next day for a new Panasonic workstation straight-from-Japan. I wrote a demo facade for workgroup software overnight from scratch using C in a windowing environment I had never used before, while the also straight-from-Japan director I had never met before slept on a bed on the other side of a huge hotel room. But even then, I would have said that kind of effort was not sustainable, even if it was fun to do it once. I’m lucky I did not have a car accident driving home from New York City after the commercial was done filming the next day and, worse, after we all went out to a wrap up celebration dinner at some Japanese Sushi restaurant.

Still, I can see value in a variety of face-to-face coding experiences. And obviously, some coding tasks like, say, new feature development or rapid proof-of-concept demos or testing GUI variations may be better matches to meetups than, say, complex refactoring tasks or complex debugging tasks or methodical translations from one language to another. Face-to-face meetups are obviously also going to help with more social programming interactions like pair programming, mentoring, or training.

And certainly the average corporate on-site setup can be filled with continual distractions too, with random people walking by, random questions called across the room, telephone chatter (like if a co-worker has just had a house built, as at one place I worked on-site), and so on.

It may be the case that some complex tasks like some refactoring tasks can indeed be done better through pair programming with two sets of eyes (or ears) on the code changes (almost regardless of the surroundings) than by one person in a familiar home office. However, pair programming (as opposed to side-by-side team programming) was not mentioned as a meetup aspect by Scott Berkun (that I recall at least — or maybe he just took it for granted?). I’d be curious to learn more about whether people at Automattic even do pair programming? I sometimes would do pair programming with a remote colleague sometimes via a join.me screenshare when working on complex code for NBCUniversal (work mentioned later).

Could the need for some work-focused meetups reflect a need for better tools?

Also on meetups, while I am all for face-to-face time for team building even if travel is needed, it might be worth reflecting on whether better software or better remote pair programming support (or even programming language or architectural choices) might make Automattic programmers more efficient and productive so such physical meetups could often be more about playing together, learning together, and team building rather than coding side by side. While meetups make sense for a variety of reasons, if teams have to physically get together for marathon sessions to make architecture/refactoring decisions or to make comprehensive project plans (as Scott Berkun described), that suggests to me something is lacking in networked tooling (like virtual whiteboards, better teleconferencing options, or whatever).

For example, Scott Berkun writes about using Skype a lot. It’s been my experience in working in distributed teams that Skype voice calling in particular is usually difficult to use for a fine-grained technical discussions or developing comprehensive plans because of lag (compared to using POTS telephones) because the lag makes it hard to interrupt or ask questions. Even just one person calling in to a regular phone-based teleconference via Skype can mess up the rhythm of the entire conversation (cell phones can also have this laggy effect sometimes, but generally much less so). IRC text can indeed seem preferable to Skype voice conferences — but landline telephone-based teleconferences without lag are a different experience altogether — even internationally (if not on a satellite link). Automattic could be a great test bed for all sorts of experiments on improved networked collaboration tools (including perhaps, say, some new approach that might somehow make the best of laggy Skype-like voice chat possibilities without breaking conversational rhythms for when non-laggy landline teleconferences are infeasible). If we could figure out together what could be added to WordPress (either directly or as a companion service, even a telephone-based one) to help address this need for improved fine-grained networked coder collaboration, Automattic might have yet another great product to provide hosting for — or maybe get a cut of the revenue from if it involves a third-party like Global Crossing. :-)

What do you feel meetups could productively focus on? Left vs. Right brain thinking?

It’s been demonstrated (see for example, The Master and His Emissary) that great products tend to come from alternating periods of relaxing playful expansive creative whole-oriented “right brain” thinking with other periods of task-focused narrowing plodding detail-oriented “left brain” thinking. A big challenge for people who take naturally to computing is to balance both those modes of thinking (and right now, this essay reflects an expansive phase by me which can both be hard to get into and hard to leave, even as at Automattic in practice I’d eventually cut this essay into bite-sized actionable focused prioritized pieces such as ultimately produced NarraFirma if you look at items moved from NarraFirma’s TODO list to NarraFirma’s DONE list.).

By the way, for an interesting take on this left brain / right brain balance issue in relation to “Objectivism”, see Michael Prescott’s “Shrugging off Ayn Rand” where he says:

To see why, let’s take a step back and look at Ayn Rand’s philosophy from a different perspective. The best one-line critique of Objectivism I ever heard was leveled by a friend of mine, who said, “Ayn Rand was the ultimate spokesman for the left hemisphere of the brain.” The left hemisphere is conventionally associated with methodical reasoning skills, which are obviously a crucial part of what it means to be human. But there is also the vast spectrum of thought processes linked to the right cerebral hemisphere – the ability to think in terms of nonverbal associations, symbols, metaphors, images, poetry, myth, music, shapes, and holistic patterns, and to generate intuitive insights that cannot be reduced to linear reasoning. There is the mysterious realm of emotions and personal interactions, the unconditional love of a parent for a child, the willingness to sacrifice for a cause greater than oneself. Little of this finds its way into the sterile, airless terrain of Rand’s philosophy.

Objectivism’s narrow emphasis on reason and logic might be helpful to a certain kind of person – a person whose right-hemisphere functions are overdeveloped, who is intuitive to the point of being flighty, spontaneous to the point of being scatterbrained, someone who is wildly emotional and has difficulty with linear reasoning. For this sort of individual, Objectivism might serve as a helpful counterbalance and corrective. The problem is that this is precisely the sort of person who will be turned off by Rand’s philosophy and will be unlikely to pursue any serious interest in it. The person who is drawn to Rand is typically someone who is already highly left-brain dominant, somebody with a propensity to think rationalistically, to suppress or distrust emotions, and to render moralistic judgments in terms of absolutes, without acknowledging ambiguities. I certainly fit this profile. When I became an Objectivist, my pre-existing tendencies were encouraged and amplified, and instead of becoming more well rounded, which is the kind of personal development I needed, I became even more one-sided.

If I had been thinking more “right brain” that last weekend my father was alive, I might have gone to see him despite obstacles, rather than going back to pleasurable “left brain” coding finishing some project and just assuming my father was going to be around the next weekend like every weekend before that. Finishing the project could have waited one more day, and it probably would have been even better for it; looking forward to doing something can even make it more pleasurable overall. That’s an extreme example, obviously, but our lives are still made of a succession of such small-at-the-time-seeming left/right choices. So, how can Automattic use meetups and other practices (including “creeds”, “mottos”, “principles”, or “practices”) to promote a healthy balance of both left-brain and right-brain thinking?

Better tools that increased “left-brain” productivity when working remotely might make it more possible for teams to play and learn together and focus on “right brain” activities more during team building meetups. One can imagine networked tools for enhancing right brain thinking as well (like for nonjudgmental “brainstorming”), but those are likely much harder to create. To encourage people to enter a playful state of mind when face-to-face, there are certainly many alternatives to hanging out together in random bars to the wee hours of the morning (as good as that may be in some cases). Also, Scott Berkun and Team Social and many others may enjoy late nights at bars and find that relaxing at certain points in their lives (especially their 20s), but as pointed out in the Amazon comment above, not everyone does. Some other alternatives for meetups are discussed in the next section.

How do you think meetups will likely change over time at Automattic?

The average age of Automattic employees will likely increase over time given such an admirable employee retention rate. As time goes by, more Automatticians will likely commit to life partners, have children, take on caregiving responsibilities for aging parents, and/or take on local community responsibilities as volunteers. If so, choices of meetup venues are likely to change to accommodate changing interests and priorities. I’d suspect that is already starting to happen as Automattic turns ten yeas old, and that it will almost certainly happen by the time Automattic turns twenty.

As time goes on, likely there will even be calls for family-friendly venues for Automattic meetups (including perhaps with SAS-like onsite childcare or child-oriented activities) so Automattician’s life partners and children can interact. A shift to a family-friendly focus happens at almost any big company that has community-building company-wide get-togethers, and it also happens at college reunions or homecoming events. As a child, I had a great time at “homecoming” events at Farmingdale State College where my father got an associates degree in his fifties (tip — do not have events at meetups where kids can win goldfish or other pets that require many years of care that child may not have contemplated before tossing a quarter into a bowl. :-) Princeton is especially worth studying in that regard as well-run reunions are a likely a big reason for Princeton’s success in alumni engagement, and if you look at the different class years, you will see the activity focus change from big all night social parties with top bands to more educational and family-friendly events (but still with great music).

Here are a few of the many alternatives to bars and bar games for creativity-promoting meetups. A team might benefit from a long weekend together at the Humor Project’s Humor/Creativity Conference (and which I’ve gone to on my own to great enjoyment and personal growth). Such an experience might do wonders for team building and future productivity — even if not a line of code was written at the time or even if no to-do list was assembled. There are clown schools, trapeze schools, wilderness survival training courses, space camps, theme parks like EPCOT, seriously defensive/evasive driving courses, tank driving courses, science fiction conventions, gaming conventions, technical conferences other than about computers (think of them as market research on potential WordPress users), SCA conventions, hands-on craft learning like canoe making, first aid training, nature tours, city tours, special museum tours, major planetarium tours, martial arts camps, physical fitness camps, sailboat classes, themed cruises, health getaways run by various celebrity doctors, and cooking short courses (to name a few). Just traveling to places like Athens and actually spending days touring museums and historical ruins together, totally away from computers, might be perspective-promoting and team-building for Automatticians who otherwise are glued to laptops. Or how about a long weekend together at music training/jam sessions with expert music educators? Or, as a stretch, not for everyone, how about a week together at a medically-supervised water-only fasting retreat center like “True North Health Center”? (I’ve done water-only fasts myself and found them interesting experiences, and supposedly fasting can help regenerate hippocampus/memory cells among other health benefits, which may be why fasting became a part of every major religion.)

There are also potentially team-building volunteer opportunities that could be done together by an Automattic team like Habitat for Humanity home building, Isles Inc community garden construction or planting, Patch Adams Gesundheit! Institute hospital construction, and so on (even just cleaning up roadside or wilderness litter as a team). Those volunteer activities might be even better than self-focused “educational” activities for building new shared memories of shared physical accomplishments (given human psychology of teams and tribes is probably still linked to doing physical activities with visible results together, like Amish barn raisings). Some of those volunteer efforts like house building or litter cleanup might make for good publicity too, incidentally, but if they are done mainly for publicity value they may lose much of their personal and team meaning.

All of these activities might help people get out of the mental ruts we all can fall into.

So what did you do next after your father died?

After my father’s death, we decided to move closer to a sister upstate so we could ultimately place my mom in a nursing home where she could get multiple visitors. So we moved to a rural area in upstate New York, in the Adirondack Park, with a lower cost of living and a lot of trees (about fifteen acres worth). It was, however, far from most on-site programming jobs. We had hopes of living frugally and creating a foundation (pointrel.org) to write more free software and content.

However, life had other plans. In 2003 our son was born. (The first morning after we moved into our house, we saw a deer give birth outside our living room window. It must have been a good sign.) So our relative frugality, instead of giving us time to make a lot more free software full-time, made it possible for both of us to spend more time with our son.

With my wife’s career doing well, and a programming recession in full swing after the dotcom crash, I ended up doing a “Watching the Wheels” / “Dan Ingalls’ Hotel Years” opt-out for a time. I became a stay-at-home Dad for several years, helping raise and homeschool our son during much of the 2000s. My wife worked (with some gaps) with IBM Global Services Research and then eventually as Director of Research for a spin-off company called Cognitive Edge (which got a bunch of government funding related to StoryML and other related ideas in the area of narrative and sensemaking, discussed more later).

How did being a stay-at-home Dad work out?

I could probably have found more regular paying work than the on-again-off-again contracting my wife was doing for IBM and Cognitive Edge. But given many factors related to our location and child care, that probably would have meant the end of Cynthia’s career, which she liked overall and found meaningful (and which also paid well, when work was available). Neither of us was that keen on bringing in someone else to raise our kid during much of the day.

Cynthia still did a lot of the childcare and housework when she wasn’t working, though, so I can’t say I was a model househusband.

What I saw was that I was doing about half the child care and domestic duties, which was far more conventionally-considered women’s work than my own father did. I still did a lot of other household tasks (like bills and car maintenance). The rest of my time I was making mostly free software and free content to help the world. Also, I was in a way helping mentor college-aged kids via the internet and Slashdot, Open Manufacturing, OpenVirgle, and similar communities. I felt we were living in a rural area because my wife grew up in one and liked it, whereas I’d prefer to live close to a town. I felt I had worked for years before full-time mostly on-site at contracting jobs in the past while she had been able to work from home on our garden simulator and have a lot of autonomy. We were making so much money from my wife’s three-quarters time work from her home office (and my occasional help with that) that we could pay all our bills while eventually also often putting away thousands of dollars a month by paying off our mortgage. So to me, things seemed good and reasonably fair.

But what my wife saw was that I was doing only about half the child care and housework, and that she was bringing in almost all the money doing sometimes stressful and difficult work, and that we were living somewhere cheap because I was not doing my share economically, and that her work on my garden simulator idea and other software was equivalent to my having brought in money from consulting. So, she felt I was way behind on my obligations of fairness in our marriage (her ideal being an ongoing 50/50 split of childcare, housework, and paid work).

That difference in perspective about fairness was a source of friction in our marriage.

Lucille: [seeing superhero auditions in her home] Oh, I don’t deserve this!
The Shoveller: I know.
Lucille: A lot of other men I could have married, Eddie – still are!
The Shoveller: I understand.
Lucille: If ONE person vomits in my pool, I’m divorcing you!
The Shoveller: That’s fair.

Did you do anything besides child care during that decade?

During that time, I worked on a variety of small independent writing and programming projects, some of which can be found linked from my pdfernhout.net site. Examples:

  • I wrote several essays related to free software, free content, and the maker movement (including a book-length essay called “Post-Scarcity Princeton” around 2008).
  • I ran a small WordPress blog for a time called beyondajoblessrecovery.org that eventually I migrated to a static site around 2010.
  • I wrote about new ways to conceive of global security and rethink the CIA in an age of potential abundance for all.
  • I worked towards constructivist educational tools in Python inspired by Squeak and eToys (called PataPata). PataPata essentially turned Python into a full-fledged Self-like prototype-based Morphic-like development environment (this was before I got into JavaScript, but I had learned about prototypes using NewtonScript). I eventually left PataPata behind for various reasons. (Here is a video by Francois Schnell as an introduction to PataPata that captures its essence.)
  • I worked on new versions of my Pointrel triplestore.
  • I was involved in the OpenVirgle and Open Manufacturing news groups, discussing how open source ideas might be brought to the physical world (including through building on my 1999 proposal for OSCOMAK). Here are some thanks to various people in my life who have helped me along the path to good work which I put together when working on a later version of the OSCOMAK site which was an active community for a short time as a Halo Semantic MediaWiki.
  • I was on an internet radio show about post-scarcity economics a couple of times.
  • I made some educational videos related to post-scarcity and economics.
  • I put together a list of about 100 ways to move beyond a jobless recovery.
  • I posted way too much to Slashdot, both for my own education through discussion and also trying to help lead other software developers to better health and a better society (thankfully getting quite a few +5 comments, the occasional down-modded 0 comment, and also a couple of front page posts along the way).
  • I participated in p2presearch discussions. (Incidentally, Michel Bauwens, who leads the Peer-to-Peer Research group, and who I’m friendly with despite some disagreements about future trends, might be an excellent advisor for Automattic.)
  • I participated in some private Princeton alumni mailing lists related to parenting, non-profits, and general discussion of current events.
  • I wrote two Android apps related to music called EvoJazz and Musical Phrases (discussed below).
  • I learned a lot about health issues such as the importance of supplemental vitamin D for indoor workers and the importance of eating a lot of vegetables for long-term health for anyone; I also set up DIY treadmill workstations for both myself and my wife to get exercise while programming and writing.
  • I developed the tagline used in my email sig and elsewhere. That tagline is probably my greatest writing accomplishment (especially for conciseness). It sums up the most important single thing I’ve learned about technology and society: “The biggest challenge of the 21st century is the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.” I can thank standup/sitdown comedian Brett Leake who I met at the 2008 Humor Conference for helping me think about irony and important messages. I later used his advice to “find the irony in a situation and draw it out” in other writings like the Downfall parody linked below.
  • I can also thank the Humor Conference itself (and the husband/wife team of Joel Goodman and Margie Ingram who organize it) for helping pull me back from the sad state of mind I was in after writing Post-Scarcity Princeton, having dwelled too much on the negatives and frustrations of my time at Princeton in various roles as well as my then-current feelings about the place as they interacted with some current events, given that Post-Scarcity Princeton arose from an intense “Dark Nights of the Soul“-style exploration of three core celebrated Princeton values (and their shadows) of elitism (alienation), competition (destructiveness), and excellence (perfectionism). As in Ursula K. Le Guinn’s “A Wizard of Earthsea“, we all have our shadow though; the question is what to do about it?

More links from that time are on my pdfernhout.net website.

One of the more popular and funny things I wrote during that time was this “Downfall” parody about post-scarcity:
http://blog.p2pfoundation.net/in-the-fuhrerbunker-hitler-confronts-the-post-scarcity-economy/2012/10/13

But some of my writings about “basic income” proved mildly popular too:
http://www.livableincome.org/amillionairegli.htm

What are your musical Android apps EvoJazz and Musical Phrases about?

Towards the end of that time, mostly to try to raise some money, I developed two versions of a music design application for Android, with the simpler version (which actually came later) called “EvoJazz“, and with a more complex version called “Musical Phrases”. Both were inspired by ideas in our PlantStudio software (itself inspired by an ideas my wife and I learned about from studying evolution). They work by randomly generating variations on musical phrases, with the user picking ones they like, and then getting more variations on them.

EvoJazz Android App

EvoJazz Android App

I create a business concept that I would release the source code for each release of those Android apps a few years after each version came out, as a form of temporary “Artificial Scarcity”. That link leads to an ironic, but educationally-intended, site I set up about the idea. What I found though was that being proprietary cut me off from the open source community and feedback on the code or pull requests for improvements (which also reduced my own motivation to work on the code). I did not get much revenue given limited word of mouth from charging for the apps. I did not feel good about the whole process. And in the end, nobody still has ever asked about the source code. People were also disappointed in their expectations, since people seemed to want software that they could use like a regular note-by-note editor, not software that generated phrases for them. As a potentially paradigm-shifting application, EvoJazz just did not make its case well enough. Most of the registered users seemed to come from Japan from some reason, where maybe that idea of Evolutionary Jazz took hold more easily?

While the “temporary artificial scarcity” idea was a bust, here is another one idea related to EvoJazz. It’s been said that the Jazz Age in the 1920s could never happen today, as musicians would be suing each other when they copied from each other and elaborated on themes rather than building a musical world together. That seems a sad situation to me, even as ideas like Creative Commons licenses help address it a bit. Because copyright in theory is supposed to only cover “copying” a musical phrase, the potential exists to turn the music industry upside-down building on EvoJazz ideas. A network of EvoJazz users might all run similar software as a JavaScript-powered WordPress plugin connected through a common sever to archive and share musical phrases discovered. Such a network could independently crowdsource a free body of musical phrases usable by musicians everywhere by random variation and selection. That might help deal with what Nina Paley calls “brain damage” caused by copyright. Such a library could not easily be claimed to be “copied” since it was randomly generated (with phrases probably generated on the server to ensure they are truly randomly created with no “cheating” by a hacked client). There are no doubt various other legal aspects of copyright to consider as well though (copyright is a slippery thing these days). And also EvoJazz right now just handles one melody line of sequential music, not two or more overlapping melody and harmony lines of notes from chords of multiple piano keys or multiple instruments.

I have not had the resources or guts to go up against the music distribution cartel myself though — anticipating facing a stormy sea of legal threats and takedown notices for supposed “copyright infringement” when a randomly generated and selected musical phrase happens to correspond with a musical phrase the cartel claims ownership of. So, that idea to liberate music via Evolutionary Jazz has sat on a shelf for years. Maybe someday that idea will get dusted off when someone (or some Jazz-loving team) braver and more resourceful than I takes it on as a creative challenge? :-) But, it would still be pretty dangerous to try it in a business climate that often worships indefinite “artificial scarcity” as an economic model. :-(

[commenting On The Disco Boy’s arsenal]
Mr. Furious: What? Guns? That’s your power, you shoot guns?
The Blue Raja: There’s no theme at all here.
Mr. Furious: Weak.
The Blue Raja: At best.
[a Disco Boy wields a pipe]
The Shoveller: And who are you supposed to be, the Disco Plumber?
[they high-five each other]
Mr. Furious: See, you’ve got a chain, I would at least make it a gold chain. That’s just off the top of my head.
Mr. Furious: [to Tony C, referring to his outfit] What’s up, Tiger Lily?
Tony C.: Top of your head, huh.
[Tony C pistol-whips Furious]
Tony C.: [the Disco Boys proceed to beat the hell out of Furious, Blue Raja, and the Shoveller]

So, how did you get in bed with the defense industry?

During that decade, I also helped my wife related to her work with IBM, Cognitive Edge, DARPA, and Singapore by implementing some software related to collecting and analyzing stories. For example:

  • I ported some software from Python/wxWindows to Jython/Swing (called SenseMaker Explorer, otherwise Cynthia’s work and based on Cynthia’s research in narrative-based technologies, ~2004 via Cyberneutics, and later with Cognitive Edge, as essentially an IBM spinoff). I also helped her package that desktop software for Mac, Windows, and Linux.
  • I helped implement a multi-user Concept Demonstrator with her for Singapore’s Risk Assessment and Horizon Scanning RAHS program (also done through Cognitive Edge, 2005). That work finally paid off the last of our debt from our garden simulator years. The Concept Demonstrator had a PostgreSQL back end in Python and had a rich-client Jython front end. An overview of RAHS is the Technology Demonstration video here, and Cynthia and I wrote some of the software you see there (the SenseMaker Explorer part, although the Concept Demonstrator prototyped an early version of some of the rest).
  • I wrote (with Cynthia) some web software (called SenseMaker Collector, 2007, again for Cognitive Edge) in Python/Django for collecting stories using web-based survey forms.

Two of those projects (SenseMaker Explorer and Collector) formed the core offerings of Cognitive Edge as proprietary software (although they offered some other packages written by others). Almost a decade later, they are still offering almost essentially the same software (showing that our implementations stood the test of time).

We tried for years to get Cognitive Edge (CE) to open source the code we wrote, which was mostly funded with government tax dollars, but ultimately to no avail. I feel that this was unfortunate, especially for CE, as otherwise I think the market for training for that software (a big part of CE’s revenue) would have expanded greatly. Also, Cynthia and I would have been very motivated to continue to expand the software in various ways, even at reduced compensation. But such was not to be. There was some success getting CE to open source some sensemaking “methods” though (which would be hard to claim control of anyway), although the full descriptions of some of those methods seem to have disappeared from easy access on the CE site.

How did you respond to feelings about this example of tax dollars funding proprietary software?

Years later, I wrote a couple of essays related to the general issue of tax dollars going to fund proprietary content. Thankfully policies on that are slowly changing, with more and more government and foundation grants requiring that results be made available as open source.

The first essay on this is a shorter version of the second:
An Open Letter to All Grantmakers and Donors On Copyright And Patent Policy In a Post-Scarcity Society
On Funding Digital Public Works

I’ve also called for the US Government specifically to fund open source public intelligence tools for use by global community to make the world a safer and happier place (to not much response, but maybe someday).

Does anyone else care about open source sensemaking and public intelligence software?

I’m not the only person suggesting such things (even if my voice may be the smallest in the chorus). Both ex-CIA officer and author Robert Steele and former Defense Intelligence Agency analyst and software developer Matthew Burton as well as others have made similar calls in their own ways and sometime for somewhat different but overlapping reasons. As a Wired article on the topic says:

For three years, Matthew Burton has been trying to get a simple, useful software tool into the hands of analysts at the Central Intelligence Agency. For three years, haggling over the code’s intellectual property rights has kept the software from going anywhere near Langley. So now, Burton’s releasing it — free to the public, and under an open source license.

Burton, a former Defense Intelligence Agency analyst and software developer, speaks today at the Military Open Source Software Working Group in Virginia. It’s a gathering of 80 or so national security tech-types who’ve heard a thousand stories about good ideas and good code getting sunk, because of squabbles over who owns the software. …

“The Department of Defense spends tens of billions of dollars annually creating software that is rarely reused and difficult to adapt to new threats. Instead, much of this software is allowed to become the property of defense companies, resulting in DoD repeatedly funding the same solutions or, worse, repaying to use previously created software,” writes John M. Scott, a freelance defense consultant and a chief evangelist in the military open source movement. “Imagine if only the manufacturer of a rifle were allowed to clean, fix, modify or upgrade that rifle. This is where the military finds itself: one contractor with a monopoly on the knowledge of a military software system.”

Everyone needs some security. The question is, how do we go about ensuring adequate security for all (even our current supposed “enemies”) in non-ironic and non-counterproductive ways? That’s a difficult question to answer whether in relation to digital security or physical security. It is an issue that IMHO needs a lot more exploration, discussion, experiment, and deep thinking — as well as deep feeling.

So, are you evil because you and your wife helped DARPA/IARPA with Genoa II and Singapore with RAHS?

The politics surrounding things like Singapore’s RAHS or various US DARPA/IARPA programs are complex (and yes, I wrote the 3D landscape viewer tool displayed in that Wired article and some of the rest). However, the intent of the tools we worked on for both the US Government and Singapore was to help decision makers and analysts see issues from multiple perspectives, whatever other priorities some might have in using such tools or reporting on them.

The fact is, if no one with a conscience thinks broadly about security issues, what sort of security are we going to end up with given a lack of cognitive diversity?

Still, having studied ecology evolution, it was also clear how dangerous and wasteful arms races can get. It remains a tremendous challenge to create secure situations without causing arms races. Perhaps open source is part of the answer to that?

Having read some of Langdon Winner‘s writings on technology and politics (way back around college), I would agree that the choice of what tools to make is important in shaping our society — especially given various consequences that flow from what is easy to do. We should make good choices about what to implement and what not to implement (including considering cultural context and likely immediate use) — just as we should also work though legal issues about using such tools. That said, even a plowshare can be used for no good in a variety of ways if people are so inclined.

The Yin/Yang symbol reflects this by each of the two major parts of it not being purely black or purely white. Still, these issues remain controversial. It was saddening to me to in some sense lose some “purity” by getting involved more closely for a time with the defense industry.

It was also career limiting for Cynthia that we chose to do that work at arms length through another company to try to stay a bit more “pure” instead of Cynthia being a principal directly contracting with DARPA. That was a choice which would come to be a bigger issue down the road as it ultimately affected copyright ownership of the work we did then, leading it to be proprietary instead of open source.

Whatever one might feel about those past efforts, or potential alternative paths to funding for such research back then, how can we make the most of the related ideas now for the public good?

Many businesses and other organizations (including non-profits) have found such tools useful to support their missions, often with Cynthia’s help to ask the right questions and to make sense of the results. For example, Cynthia used such software tools and her expertise to help with projects in Canada on fighting forest fires and improving youth soccer. She also helped with a project on the future of volunteering in the community care sector in New South Wales, Australia (based on real-life stories collected from volunteers, their managing organizations, funders and policy makers).

A key finding from that study on volunteerism was: “The surprising thing here is the centrality of emotional needs. It seems that when emotional needs are prioritised, volunteers perform critical roles, feel recognised, maintain control, and pursue options; in effect, the system works.” However, different types of volunteers had different emotional needs. That entire document (put together by others, but based on her work and her lengthy written report) might worth reading by people at Automattic to get general insights into how to make the WordPress community of volunteers even stronger.

It might be worthwhile to do a similar NarraFirma project that was specific to the WordPress community and volunteerism, since communities and volunteers can differ in their specific priorities and needs. There is nothing like stories from your actual community to make abstractions like “emotional needs” feel more real and to connect those abstractions with specific actionable recommendations often emerging from patterns in the collected stories.

So how do these ideas about sensemaking and public intelligence relate to Automattic and WordPress?

Some of the ideas in that DARPA and RAHS work (including some ideas shown in the RAHS videos) are ideas I would like to bring to WordPress as JavaScript-powered plugins and such. NarraFirma was a small start down that road (as were Cynthia’s Rakontu and NarraCat projects before, and as are my own Pointrel and Twirlip efforts). Such tools could eventually help WordPress-powered communities (including Automattic) do their own collective sensemaking on a variety of issues they are interested in, whether technology risks and opportunities, political trends, investing wealth, improving health, or internal challenges like from rapid growth.

While Automattic has pursued a mission to democratize publishing with WordPress and has had much success with it, I’d suggest the opportunity exists now to build on that success to make the web a better place by democratizing sensemaking through open source public intelligence tools such as ex-CIA officer Robert Steele advocates (used both by communities via public crowdsourcing and by individuals and small teams in their own private ways). Such crowdsourced public intelligence work might including democratic sensemaking by communities about what government is up to in their name. Such software could be as a step towards a bi-directional Transparent Society and new forms of 21st-century participatory journalism — in contrast to the one-way opaque information flows and the corporate journalism we have so much of now.

Is this really worth the risk of Automattic getting involved in more than web publishing?

As I suggested five years ago in a response to a call for ideas by the US government, if we do nothing to help individuals and communities make sense of their own situations, we will almost certainly fail as a society given many existential risks; it is only by taking some action that we have some chance of success:

Now, there are many people out there (including computer scientists) who may raise legitimate concerns about privacy or other important issues in regards to any system that can support the intelligence community (as well as civilian needs). As I see it, there is a race going on. The race is between two trends. On the one hand, the internet can be used to profile and round up dissenters to the scarcity-based economic status quo (thus legitimate worries about privacy and something like TIA). On the other hand, the internet can be used to change the status quo in various ways (better designs, better science, stronger social networks advocating for some healthy mix of a basic income, a gift economy, democratic resource-based planning, improved local subsistence, etc., all supported by better structured arguments like with the Genoa II approach) to the point where there is abundance for all and rounding up dissenters to mainstream economics is a non-issue because material abundance is everywhere. So, as Bucky Fuller said, whether is will be Utopia or Oblivion will be a touch-and-go relay race to the very end. While I can’t guarantee success at the second option of using the internet for abundance for all, I can guarantee that if we do nothing, the first option of using the internet to round up dissenters (or really, anybody who is different, like was done using IBM [punched cards and tabulators] in WWII Germany) will probably prevail. So, I feel the global public really needs access to these sorts of sensemaking tools in an open source way, and the way to use them is not so much to “fight back” as to “transform and/or transcend the system”. As Bucky Fuller said, you never change thing by fighting the old paradigm directly; you change things by inventing a new way that makes the old paradigm obsolete.

When Alan Kay and his team at Apple create Squeak, his stated intent was to give people a tool they could use to bootstrap something even better. He said essentially that some Squeak developers though he had made Squeak as a final amazing tool for them to use, and thus such developers thought that any criticism of Squeak was criticism and disrespect of their hero, Alan Kay (and the rest of his team). He was unhappy that some Squeak developers would get so defensive about Squeak and Smalltalk as-it-was when other developers made suggestions for major improvements, rather than use Squeak as a vehicle to explore the fundamentals of new computing as he had intended, which was disappointing to him. Of course, there is nothing wrong with benefiting from Squeak as it is, but standing in the way of significant change to it missed Alan Kay’s bigger picture intent of exploration and experiment. Doug Engelbart had similar attitudes with his Augment system talking about a co-evolution of people, processes, tools, knowledge, and, hopefully, wisdom.

WordPress has democratized publishing on the web for many people to great benefits globally. WordPress should of course keep doing that as well as possible as that’s an important mission. But supporting that mission does not completely exclude other areas of growth for WordPress and the community that surrounds it (including transformative growth, like a caterpillar into a butterfly). Why not consider WordPress today as a collaboration-supporting tool that people can use and build upon in various ways (especially via JavaScript-powered plugins such as NarraFirma) to co-evolutionarily design and bootstrap something even better for pressing words together to make the web a better place? Why not continue to improve WordPress in the service of democracy (or other healthy and humane social processes, such as chaordic ones, stigmergic ones, or so on for whatever balance of meshwork and hierarchy is appropriate for specific cultural situations)?

No doubt there will be all sorts of difficulties down this path. However, many experienced programmers, designers, community leaders, content creators, legal experts, system administrators, testers, and others have moved up the curve of ability, and to sustain a sense of flow such people actually need hard fun creative challenges. And of course, morally, those creative challenges should ones that promote the health of the surrounding community (even as moral debates over what constitutes health for whom given various differing priorities are their own challenges).

The Shoveller: Lucille, God gave me a gift. I shovel well. I shovel very well.
Lucille: Honey, you shovel better than any man I’ve ever known, but that does not make you a superhero.
[the Shoveller starts to say something, but Lucille cuts him off]
Lucille: No, listen to me. You’re a good husband, and a good father. But that’s all. Nothing more.
[she walks off, a small boy hugs the Shoveller’s leg]
Roland, The Shoveler’s Son: I believe in you, Daddy!
Lucille: [calling] Roland, do NOT encourage your father!

Matt Mullenweg called for other companies to dedicate 5% of their people to improving WordPress. What could the web (and the world around it) be like after a decade if 5% of WordPress development effort each year went into better WordPress tools for sensemaking and public intelligence? Would the likelihood of the WordPress community (and its physical surroundings) being an even more joyful thriving place be higher or lower after such an effort?

Of course, I’m not asking for 5% — well at least not at first. :-) I’m just asking for my own time, and the time of anyone foolhardy enough to voluntarily join me from the larger community as a free and open source effort. :-)

The Shoveller: We’re not your classic heroes. We’re the other guys.

Is better technology the whole answer?

I like writing software tools, so I am biased in thinking they can help a lot with various issues. But they do have their limits.

Ultimately, as with Europe’s fumbling first steps towards “the right to be forgotten” and rights related to moving data across national borders (whatever one thinks of the specifics of those particular efforts) , we may need the political equivalent of zoning laws to control what is done with informational technology (same as with some other technology). Just because you can do something with technology does not mean you should.

Looking broadly at the civil rights movement in the USA, it went for over a century, and included a civil war in the middle of that, and it is still not done. And the concept of “freedom” even continues to expand in all sorts of ways such as women’s rights or gay rights and so on (or perhaps cycle instead of “expand”, if you consider large parts of North America and elsewhere on the planet used to be run as matriarchies, some with various alternative gender roles). Likewise, we may even be cycling back to the economic independence of the yeoman family farmer through the better aspects of the maker movement and the sharing economy (subject to debate). So, the struggle for freedom and empowerment (as well as defining reasonable responsibilities and healthy limits) may be never ending in all sorts of ways.

As Toni Morrison wrote:

“I tell my students, ‘When you get these jobs that you have been so brilliantly trained for, just remember that your real job is that if you are free, you need to free somebody else. If you have some power, then your job is to empower somebody else. This is not just a grab-bag candy game.”

Yet, as Robert Locke wrote in The American Conservative:

The most fundamental problem with libertarianism is very simple: freedom, though a good thing, is simply not the only good thing in life. Simple physical security, which even a prisoner can possess, is not freedom, but one cannot live without it. Prosperity is connected to freedom, in that it makes us free to consume, but it is not the same thing, in that one can be rich but as unfree as a Victorian tycoon’s wife. A family is in fact one of the least free things imaginable, as the emotional satisfactions of it derive from relations that we are either born into without choice or, once they are chosen, entail obligations that we cannot walk away from with ease or justice. But security, prosperity, and family [and I would add, health and community — pdfernhout] are in fact the bulk of happiness for most real people and the principal issues that concern governments.

So, even a simple-sounding topic like “freedom” has complex nuances. Why should “information technology” be any simpler to think through, when it involves conflicting issues like freedom versus responsibility and empowerment versus limits?

As with civil rights and other rights, thinking through how we (either as a global community or as local communities) want to use information technology within democracies (as well as other political forms arising out of explorations of various mixes of meshworks and hierarchies) may likewise be a long struggle with complex dynamics going on for many decades.

Like the eternal struggle between mildew and sunshine, or even the rise again of slavery in the Middle East discussed in current events, even after we think the issues are settled, that political struggle may perhaps be ever ongoing. And even successful social change may continue to see new horizons to progress towards. It is generally in the nature of politics to never be completely settled, including due to new generations with new ideas and their own time-and-place-specific desires needing to “learn what the words on the monuments mean” to reverse ever-present processes of decay their own way.

Radical change may even be demanded if you just want to keep things the same, :-) as G. K. Chesterton wrote about:

We have remarked that one reason offered for being a progressive is that things naturally tend to grow better. But the only real reason for being a progressive is that things naturally tend to grow worse. The corruption in things is not only the best argument for being progressive; it is also the only argument against being conservative. The conservative theory would really be quite sweeping and unanswerable if it were not for this one fact. But all conservatism is based upon the idea that if you leave things alone you leave them as they are. But you do not. If you leave a thing alone you leave it to a torrent of change. If you leave a white post alone it will soon be a black post. If you particularly want it to be white you must be always painting it again; that is, you must be always having a revolution. Briefly, if you want the old white post you must have a new white post. But this which is true even of inanimate things is in a quite special and terrible sense true of all human things. An almost unnatural vigilance is really required of the citizen because of the horrible rapidity with which human institutions grow old.

How can better tools help with the process of making better laws?

Thinking through all those political aspects of sensible restrictions on the use of technology by many communities over the long-term is another thing such public intelligence sensemaking tools might be able to help with (and my OSCOMAK project was a tiny step in articulating goals in that direction). As an example about thinking about technology and its implications, the Amish are not anti-technology so much as that they always consciously pick technologies that promote community to their own standards. Their choices may not be mainstream, but at least they are trying to be reflective on a difficult issue. WordPress-using authors have created much content about many topics (including thoughts on technology and society such as parts of this document), but we could hopefully still organize all that collective thinking better and make more sense of it in various ways

Of course, you can organize information with more text-based discussions in IRC and more blog posts in WordPress, and no doubt such activities will be important. Automattic shows what is possible with internal use of P2/O2 blogs, including just by making prioritized lists as Scott Berkun recommends. But at some point, quantitative changes of a lot more information produce qualitative changes of very different behaviors and needs as the scale change (just like how ants and elephants have different approaches to something basic like staying warm because of scale differences). As the scale of what we are doing collaboratively changes, we can then sometimes benefit from additional options of tooling, infrastructure, and architecture — even as other things like norms, rules, and incentives remain important (as Lawrence Lessig says in Code 2.0). Why not build such thinking-together tools on top of WordPress? The multi-user clustering diagrams in NarraFirma are a start, but there is so much more that could be done to help people organize ideas and visualize relationships with WordPress given the increasing capabilities of browsers like though Canvas, SVG, and OpenGL support. For example, here is a collection of “Lexical/Text” visualizations to help in imagining what other shapes one can press words into besides linear paragraphs. :-)

Was there any other software you wrote during that decade that relates to web technologies or other innovative stuff like NoSQL?

Towards the end of that decade I started playing with CouchDB, a NoSQL document-oriented database that emphasizes using JavaScript and JSON, thinking about using it for Twirlip and maybe Rakontu 2.0.

Here is a concept map from a diagramming tool I wrote for CouchDB (using Dojo GFX). That page unfortunately won’t load correctly by default in some newer browsers (like the latest version of FireFox) due to mixed http/https content which I’ve since learned to avoid (which in this case leads to Dojo not being loaded because Cloudant now forces http links to https, unless you click on the shield icon in your browser’s toolbar to permit that). The concept map explores a debate about scarcity versus abundance thinking I had participated in on the P2PResearch list and with some private email with others. Clicking on circles with a green border leads to other related concept maps. (I would later reuse that code in NarraFirma, ultimately rewriting it for D3.)

Twirlip Concept Map Screenshot Example Of Catton Vs Simon

How do you feel about your various personal projects from that decade?

None of those personal projects or writings went “viral” or really made a huge difference in any noticeable way. Still, I’d like to think that taken together they add up to a small positive difference in helping our society transcend to one that is healthier, happier, wealthier, and both more secure and more democratic.

Was it worth the effort? I could have probably made a bunch of money with the same time if I had spent it differently, even just working from home. My wife probably would have been a lot happier if I had done so because that would have fulfilled her sense of what was fair as a 50/50 everything split. And my kid would then likely have parents who gave him a college fund, instead of parents who have run up their credit cards again and have no retirement savings. Still, I can ask myself, what sort of world is my kid going to be living in twenty years from now? And would a pile of money just for him really bring him happiness and security?

My feeling is that we are facing various technological singularities, and our direction coming out of any “Singularity” may have a lot to do with our moral/social direction going into one.

Thus, given continuing exponential growth of technological capacity, I feel it especially important to do all we can now to make the world a better place that works for everyone before computer-driven technological change overwhelms us as a species. That holds true even if we can imagine how future technology (e.g. nanotech or AI-powered robotics) might easily solve many current problems.

Without social/moral growth now — such as through people learning and growing by participating in the distributed democratic publishing process WordPress brings to the web — future technology (which some few humans think they control) may see humans as blocks of carbon that happen to be in the way of programmed-in aesthetic/commercial/political goals. People do the same thing when they bulldoze a patch of trees to put in a parking lot without much of a thought for any rights of the trees themselves.

So, what caused things to change at the end of the decade?

Getting back to my history (which is intertwined with that of my wife and her work): My wife’s work with Cognitive Edge ended soon after the start of the Great Recession, partially due to big clients cutting back on outsourcing consulting services (like for help with organizational change).

Next we used some money we had saved to create Rakontu, open source software to support online story sharing. Essentially, we went from having our house within sight of being paid off early to suddenly owing more on it than we had originally paid for it as we refinanced it as it had appreciated a bit (it’s a relatively inexpensive house, so that’s not much). But that money helped cushion us during the worst of the start of the Great Recession.

Rakontu was written in Python on the Google App Engine. Rakontu was Cynthia’s work, though I helped a bit in design and implementation discussions. In retrospect, Rakontu would probably have been much more successful as a WordPress plugin (a lesson we took to heart with NarraFirma). Rakontu had some interest, but with the rise of Facebook and such, no one really wanted yet another social networking software platform, let alone one that was quirky or with limited support or scalability. Rakontu produced a lot of good ideas, but it sort of fell by the wayside. It did not help that it was implemented on Google App Engine, which originally had all kinds of technical issues.

From Cynthia’s essay at that previous link:

I spent part of last year building an open-source web application for story sharing and sensemaking in small groups. It’s called Rakontu. This was a dream that began in 1999 (when I first started working in organizational and community narrative) and has been growing ever since. I used up years of savings to do it, and I was able to build far less than I would like to build someday, but I had a grand time and I’m glad I did it. I wrapped up the project about a month ago and posted an excerpt from a lessons-learned document for the project.

In my lessons-learned document I said that I’m more interested in the ideas from Rakontu moving on than the actual software surviving as is. Since then a few people have asked me to elaborate on that statement. So I’ve reviewed and thought, and I’ve come up with a list of six pieces of advice for anyone who would like to incorporate ideas from Rakontu into their own effort to support online story sharing.

1. Support sharing over performing …

2. Build a café in a library or a library in a café …

3. Embody knowledge about narrative …

4. Build for commitment …

5. Decomplexify …

6. Pay no attention to that man behind the curtain …

Cynthia continued to take on independent consulting projects. She found it difficult to get new clients on her own, mostly because of limited travel time/money to go to conferences, and because she still did a lot of child care and homeschooling. She still does some projects, and she is great at making clients happy once she finds them (or, more often, when they find her through her blog and other work).


Back to work full-time, for NBCUniversal (2011-2014)

So, how did you get back into full-time programming work?

With money running low again, and Cynthia’s career slowing down, we decided that Cynthia would take on most of the child-raising and (secular) homeschooling, and I would look for full-time work. It took several months to find a job where I could work from home at a good pay rate, so I could still be around my kid more (saving time on a commute) and give Cynthia more time to keep her career alive at least some of the time.

So I took my most recent job as a contractor full-time for NBCUniversal for about two and a half years (2011-2014). That support made it possible for my wife to have enough time to finish an expanded edition of her “Working with Stories” book. That book was the basis for the NarraFirma software.

What did you do contracting for NBCUniversal?

At NBCUniversal, I supported NBCU’s broadcast automation software, which keeps things like the Olympics, the NHL, the NFL, Saturday Night Live, NBC News, and the Today Show on the air (among many other programs). I worked mostly in Java when developing new code, but I also spent a lot of time reading and porting legacy C++ and TCL code (and also, on a couple of occasions, debugging the legacy C++ code to at least understand why it was failing and how to work around that).

I also provided third-line support for the NBCU software. So, when something went wrong with the equipment putting out live TV feeds, first-line support on the broadcast floor would generally first do a reboot of related equipment. If that didn’t fix the issue, they’d call second-line support (essentially, NBCU staff system administrators) for advice. Second-line support would try various things on-site. If they couldn’t figure it out or fix it, they’d document the issue and ask me or the other programmers to look at it at some priority level. For third-line support, that often involved poring over multiple multi-gigabyte log files looking for unexpected patterns of behavior across multiple devices, sometimes looking for days for one issue (and not always with success). A couple of times I was invited on a “breakout” teleconference line crackling with an undertone of stress and strong emotion (but also with years-in-the-trenches experienced professionalism) to help make some tough decisions in the face of uncertainties with the clock ticking. When equipment malfunctioned or acted sluggishly, the fear was that buffered video feeds would run out and NBCU would be left with dead air. The first such teleconference call I was on was a strong motivation to learn everything I could about making fault tolerant software.

The core of the “Broadcast Automation” software was C++ and TCL. It was written by GE Research in the late 1990s. It connected the NBCUniversal corporate schedule to controllers for hundreds of playout devices streaming video (and other embedded content) across about 100 NBCUniversal channels. A later version was created in Java as partial port of the C++/TCL code, but it still had lots of bugs and incomplete areas when I started there.

Some of the other things I did during that work were:

  • I created hundreds of unit tests (and other larger tests) in order to understand what the Java code was doing (or not doing). Later, I created tests to ensure our changes to that codebase worked as expected. I became proficient at using JMockit for dependency injection into legacy code (where you often can’t rewrite code for easier testability for various reasons), and I wrote up guidelines for others on how to do that. When an issue was reported with the code, I would try when feasible to make a test case that demonstrated the failure, and then I would make the fix so that the test passed. These tests, along with some by other people, became the core of what we ran on a continuous integration server.
  • I helped to significantly increase the time between reboots needed for their equipment by finding and fixing memory leaks, race conditions, deadlocks (including from logging code!), and timecode issues in the newer Java software. I became good at using JVisualVM for profiling application memory and CPU use.
  • Some of the work included analyzing multi-gigabyte log files and improving logging code that generated them. I created a tool to visualize message flow within a complex messaging system called GEnesis/JIC underlying the NBCUniversal broadcast automation. That tool helped our team understand what was going on inside the complex legacy software which had some odd emergent properties as it moved millions of messages around.
  • I ported code from TCL and C++ to Java to implement new PC-based controllers for their TRINIX video routers and their SkyPath affiliate satellite system (both controlled via other boxes running QNX that buffered requests which the Java code talked to). When that part of the code base was deployed, it was the first time that Java code actually was interfacing directly with hardware controlling directly what was on air. That work was mostly dealing with complex not-well-documented legacy code that had been in use for about fifteen years, where mistakes could potentially cause outages costing millions of dollars per minute from lost advertising revenue. That also included the risk of a high profile mistake of, say, making Saturday Night Live a lot less “live” by switching the wrong video feed at the wrong time at 30 Rock. Thankfully, we managed those risks well as a team and so those sorts of disasters did not happen.

Eventually, much of that in-house legacy system was replaced by a commercial vendor-supplied software solution bundled with new hardware. This was done for various reasons, but one was that the original hardware for which GEnesis/JIC had been written was getting difficult to replace, such as hardware compatible with older versions of QNX that had to be found on eBay.

As that support work tailed off, I helped to migrate another legacy system called GRAM to a disaster recovery site. GRAM supports putting tweets, breaking news, and other messages into a ticker on live video feeds. The software was in Java, talking to an Oracle database. The project included decisions about how to keep the backup system in sync with the main installation elsewhere in the country. We ended up using materialized views to do the synchronization, mostly because the maintainers of the original GRAM installation did not want to make any significant changes to the original setup towards something more efficient (understandable given the circumstances of a complex system in production).

How did you work with NBCUniversal? Remotely?

Almost all of that NBCUniversal work was done from my home office. Using agile methods, our core team of about five contractors (all working through Vaudiosoft, a small consulting company started by an ex-GE Research engineer who had worked on the original NBC software), and about five NBCUniversal staffers, communicated by daily “scrum” conference calls, email, Skype, JIRA, and Confluence. We used Redmine for issue tracking. I used RDP a lot via a VPN to remote into lab workstations for testing. I used VirtualBox a lot so I could run Windows and Linux development and testing environments on my home office eight-core Mac Pro workstation with 24GB of memory and terabytes of RAID hard disk space. I also made occasional day trips to NBCUniversal offices in New York City (about four hours each way by car and train) a few times a year — mostly to be social, for project planning, to learn about some hardware things first-hand, and to see code I wrote running at the broadcast operations centers.

When working at home, I would usually get up early in the morning, put on my bathrobe, and start working in my home office. We had our daily developer scrum teleconferences at 10:30am attended typically also by one or two NBCU staff members, except for Tuesday when we would have an afternoon all hands meeting reviewing a Google Doc spreadsheet of task assignments maintained by NBCU and involving all NBCU staff as well. Usually I would shower and dress just before those teleconferences, not feeling comfortable for some reason to be on formal scheduled phone calls in my bathrobe and without pants. :-)

Invisible Boy: [after becoming invisible for a moment and reappearing naked] I’m invisible. Can you see me?
The Blue Raja, The Shoveller, Mr. Furious, The Sphinx, The Bowler, The Spleen: Yes!
Mr. Furious: Wow.
The Blue Raja: Two hands there, son.
[Invisible Boy covers up]
The Bowler: Maybe you should put some shorts on or something, if you want to keep fighting evil today.

How did you feel about contracting for NBCUniversal?

I liked the NBCU work and my colleagues, and it was a great education regarding building reliable software, almost all done while working from my home office so I could still be close to my kid. It was also ego boosting to be able to walk into any typical restaurant or bar and in a way see my efforts at work in the digital video streaming on some television in the background.

While I had some minor disagreements about technical priorities (who doesn’t), overall I was very impressed by the team and how well everything went despite a huge amount of technical debt that had piled up in the project over fifteen years or so. That was all technical debt I tried really hard to pay down like by creating unit tests, documenting various things about the code, and fixing bugs and so on, and to some extent succeeded. :-)

Two of the developers (the lead and another retired GE engineer) were older than I was (much older in one case), as were a few of the NBC staffers. And of course, there were younger staff I worked with closely who were still going up various learning curves on the legacy technology as well as new technology coming in. These were all NBCU staff working in situations where pressing the wrong button or pulling the wrong cable could have very serious highly visible repercussions. I learned a lot from them all in various ways.

It was amazing to think I was working with the guy who invented the remote control (well, one form of it), another guy who had created the first commercial CAT scanner and NMR scanner and a bunch of other great stuff at GE, and another who had worked on the original GEnesis digital video project (the first major network to go all digital, launched with the last Seinfeld episode). I was also getting to see what it was like to be someone (my official NBCU host) who had the steel somethings to be in charge of keeping 100 channels live on the air 24X7 despite all sorts of issues that would come up — but was still a nice guy, even under stressful situations (which included him once rushing back into the city to move a piece of test lab equipment onto the operations floor to replace a failed component while video buffers were running out). Plus, my first day going out with the team at 30 Rock for pizza, the testing lab manager (near retirement) probably saved my life by quickly pulling me back when, jabbering away, I carelessly walked into traffic not noticing the light had changed, my not having been in New York City for many many years. Thanks, Bob! I used to joke, but half seriously, that he deserved all the credit for anything I did at NBCU. :-) I was impressed too by people at IBM Research, but somehow, these NBCU people were just as amazing, but in different ways. All were generally more performance oriented in the “D” sense of Research & Development, plus the on-site NBCU staff was also more “A” as in actually “applying” the ongoing “D” in a specific context and being personally “accountable” for it, including in a very visible way. While I knew various NBCU staffers (including operators I met) could not do my job with the same level of skill (requiring a lot of concentration and reflection, so akin perhaps to symphonic musical composition), I knew I could not do theirs as well either (requiring at times a lot of split-second decision making and precision performance without do-overs and without layers of testing safety nets, so more akin to live musical performance).

However, as much as everyone was very likable and the work was challenging, I had mixed feelings about working for the mainstream media broadcast industry. At home, where we have about 5000 books, we did not even have TV set up for broadcast reception (although we had a physical TV for use with video games, especially for the Wii, and also to watch carefully chosen DVDs). We did not want our kid to be around a random stream of video media at a young age, and my wife and I both felt better and bought less junk without watching TV (although we did selectively watch a lot of YouTube videos, Netflix, and other such video media). Even as I feel digital video potentially makes all sorts of wonderful things possible educationally (as with YouTube), at NBCUniversal I also knew I was essentially helping prop up a dysfunctional status quo of mainstream US society that was broadcasting “supernormal stimuli” and keeping people locked into watching via their orienting/startle reflex.

Mr. Furious: Okay, let’s all be little automaton droids and believe everything we hear on TV.

There were some shows I thought very good, like (overall) Dr. Oz (even some of the commercials on it). Comedies like “30 Rock” or “Saturday Night Live” can be of some health benefit as well to relive stress and reset “ego depletion“. They even make some good political points sometimes. Science fiction shows on TV like Battlestar Galactica on the SyFy channel helped people imagine alternate futures; NBC in particular had been the original funder of Star Trek, a positive vision of the future that has inspired all sorts of people to do all sorts of good things. TV even had a role to play in the past for moral progress related to civil rights, even if now it may often involve mostly amusing ourselves to death as a pleasure trap.

For all the commercialism and passive spectating of the Olympics, helping keep the Olympics on the air still felt worthwhile, in particular, given the symbolic peaceful global unity of the Olympics as a way to compete in a non-violent way. The Olympics also reminds us what the human body and spirit was capable of. Other sports like hockey or football provide some social glue for common ground to discuss at parties and such in a playful way, even as I’m no big fan myself of spectator sports.

And the revenue from my work was making it possible for my wife to finish a new expanded version of her free book to help people collect and make sense of their own stories (as well as keep homeschooling/unschooling our son while we paid back debt from doing our previous independent projects).

Frankly, just about every job in our crazy society has some big compromises involved somewhere because the system is a whole interconnected network and interactions (ask me sometime about any particular job, and 95% of the time I predict I can come up with ways it is problematical in some ways). I also kept telling myself: “At least it’s not Fox News. At least it’s not Fox News.”

But all that is, admittedly, justifications for a choice made out of a bit of desperation in a society without a basic income, a stronger gift economy, better planning, and/or advanced means of subsistence. Sad to say, even if it had been Fox News, with our cash/credit at an end, I probably would still have taken the job to feed my family and pay our bills and done an OK job until I could find something else. Although, I would most likely not have gone the extra mile for Fox like I did for NBCUniversal, which is less bad than Fox in several ways. But, probably NBCU is not better than Fox in all ways, given the somewhat arbitrary way various issues are divided between US political parties, where NBCU leans Democratic and Fox leans Republican. For example, as a homeschooler, I have to like at least some of what the Republican party stands for in relation to school choice.

Both broadcasting companies are also still highly commercial and generally promote mainstream corporate messages of those who can afford to pay a lot as opposed to promoting the voice of a diversity of individuals (as opposed, to, say Automattic and WordPress, which while it may still support NBC Sports as a VIP account also supports millions of other voices). There are increasing rich-poor divides in our country, which were barely reported on for decades in the mainstream media until Occupy Wall Street. Chomsky explains why that would be the case in “What Makes the Mainstream Media Mainstream“. There is an increasing disconnect between the “news” and what really matters in most people’s lives — along with limited reporting on what abundance for all is possible with today’s technology, or even yesterday’s.

In some ways, I was being paid to work on weapons of “mass distraction” (and, to a lesser extent, “mass deception”).

What do you mean by “weapons of mass distraction”?

Educator Neil Postman wrote “Amusing Ourselves to Death: Public Discourse in the Age of Show Business” in 1985. According to Wikipedia:

Postman distinguishes the Orwellian vision of the future, in which totalitarian governments seize individual rights, from that offered by Aldous Huxley in Brave New World, where people medicate themselves into bliss, thereby voluntarily sacrificing their rights. Drawing an analogy with the latter scenario, Postman sees television’s entertainment value as a present-day “soma”, by means of which the citizens’ rights are exchanged for consumers’ entertainment.

The essential premise of the book, which Postman extends to the rest of his argument(s), is that “form excludes the content,” that is, a particular medium can only sustain a particular level of ideas. Thus rational argument, integral to print typography, is militated against by the medium of television for the aforesaid reason. Owing to this shortcoming, politics and religion are diluted, and “news of the day” becomes a packaged commodity. Television de-emphasises the quality of information in favour of satisfying the far-reaching needs of entertainment, by which information is encumbered and to which it is subordinate.

In some ways, that work I did might be even worse than, say, Ted Taylor’s early work on nuclear weapons. I met Ted a couple of times and we also talked by phone about ideas related to sustainable technology and small communities. But years earlier he used to design nuclear weapons. He said he broke down in tears when he finally visited Red Square in Russia, as it was where he used to put his compass point when computing the effects of yields for new bombs. It as only then that the enormity of what he had been doing struck him.

But still, just because something like nuclear energy or digital video can be used badly does not entirely make it 100% bad. After all, every car and tank of gasoline is as John Taylor Gatto says a potential weapon, but they are rarely used that way.

Nuclear radiation saves lives all the time through medical diagnostics. It also has been used to learn about materials (including DNA) and ensure pipes wont leak. Even nuclear bombs might have productive uses, like to drive interstellar spacecraft.

And likewise you don’t need nukes to destroy a city. Sadly, humans have been doing that for thousands of years with whatever tools and techniques were available. The Wikipedia page on the Rotterdam Blitz shows what my mother’s hometown was made to look like when she was a teenager in Rotterdam during WWII, and that was pre-nuclear even though it looks not that different in that picture than Hiroshima after it was nuked (but without the radiation). My mother’s family home was burned up in the German invasion, but she said it was an accident by the Dutch defenders. And things got worse from there for her, including seeing a relative die of starvation. Talk about growing up with a mother with food issues… :-)

Digital videos streamed by a few big broadcasters (without a “fairness doctrine“) may be problematical for democracy. Digital videos streamed by millions of WordPress sites might be beneficial for democracy. Was I being well paid to be a guard? (And thankfully without the bonus PTSD)? Or was I doing a public service to keep people healthy and laughing? Or both?

Perhaps only weapons of mass instruction (WMIs) (developed in Prussia for military reasons) may be more dangerous than weapons of mass distraction/deception like I worked on at NBCUniversal? That is because WMIs can lead to all the rest (including the use for whatever crazy-but-logical-sounding-to-some-at-the-moment reasons of whatever weapons of mass deception and mass destruction are readily available, even just radios and machetes).

As John Taylor Gatto writes, the word “public” means different things in practice in a “public” library vs. a “public” school. He said:

For some reason libraries are never age-segregated, nor do they presume to segregate readers by questionable tests of ability any more than farms or forests or oceans do. The librarian doesn’t tell me what to read, doesn’t tell me what sequence of reading I have to follow, doesn’t grade my reading. The librarian trusts me to have a worthwhile purpose of my own. I appreciate that and trust the library in return.

Some other significant differences between libraries and schools: the librarian lets me ask my own questions and helps me when I want help, not when she decides I need it. If I feel like reading all day long, that’s ok with the librarian, who doesn’t compel me to stop at intervals by ringing a bell in my ear. The library keeps its nose out of my home, it doesn’t send letters to my family, and it doesn’t issue orders how I should use my reading time at home.

Likewise, WordPress and the larger “public” web around it provides an alternative way to supply information than “public” mainstream media (even compared to National “Public” Radio and the “Public” Broadcasting Service). A big difference is “on-demand” as opposed to “just-in-case” learning. But there are other differences as well relating to democracy and publishing (like who is supplying the information, who can censor it, and who can comment on it in-line with the original).

Although, is it also possible the world wide web is the greatest potential “public” distraction yet invented? Or worse? Certainly the web could potentially be used to identify and eventually trap dissenters to detain them and worse, as has been done before with information technology. Hopefully the benefits of the “public” web outweigh the costs and risks already. But if not, perhaps the benefits of the web could be made to outweigh the costs by creating even more great open source WordPress plugins and promoting socially healthy web usage habits? :-)

Did you work on any personal projects during that time contracting with NBCU?

With those sorts of thoughts about the potential implications my paid work weighing on my soul, on weekends I sometimes still worked on ideas related to my Pointrel triplestores and also “Twirlip” (all on GitHub) with GUIs in JavaScript and Java, using PHP, Python, and Node.js for back-end storage. I kept trying to make a difference myself by working towards public intelligence tools and maybe a new version of my wife’s Rakontu software (some ideas from which made it into NarraFirma). However, any time spent on stuff like that was at a cost to family time, until we decided that at least on Sunday afternoons no one would use the computer. You can see that trend persisting on the NarraFirma GitHub punch card for 2014-2015.

One such project of my own in JavaScript and PHP is Pointrel20130202. It has a running install here. The following screenshot is a concept map made in it:

Twirlip Concept Map on Health

Pointrel20140331 is one of my better efforts from then in JavaScript as a simple in-browser development environment that stores changes using IndexedDB (intended to bootstrap even better things). If you paste the below code into the text panel there, and pick “View Below,” you’ll get the classic FizzBuzz output in a panel below the editor:

<pre id="output"></pre>
<script>
// "Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”."
var result = "";
for (var i = 1; i <= 100; i++) {
    var text = "";
    if (i % 3 === 0) text += "Fizz";
    if (i % 5 === 0) text += "Buzz";
    if (!text) text = "" + i;
    result += text + "\n";
}
document.getElementById("output").innerHTML = result;
</script>

Were there any open source opportunities you passed up to contract with NBCU?

Heartbreakingly, the day after I accepted the NBCUniversal contract, but a couple days before I started, I finally got an offer to do mostly open source development for the New York State Senate, helping them improve CiviCRM (PHP/Drupal) and other software like Open Legislation (Java). I had been impressed by the team and what they had previously done to increase transparency in the NY State Senate, including making the text of new laws available to all people for free, instead of interested parties previously having to subscribe to an expensive legal reporting service. While that regular job would have paid only about half of what the NBCUniversal contract offered, I might have been able to do some of these Twirlip and Rakontu (and eventually NarraFirma) ideas there eventually as part of a mission to increase a productive back-and-forth between legislators and their constituents. And in any case, they were a great bunch of people committed to open source, and helping improve CiviCRM just by itself would have helped thousands of non-profits globally (including even the Free Software Foundation, which uses it). They were also people who did not hold Twirlip and similar ideas against me too much since we discussed them. :-)

But, while we could have maybe lived on what they could pay (although not paying off debts) — especially if Cynthia brought in some substantial income again somehow — the job had to be done entirely on-site in Albany, NY. While I had no big objections to working on-site there as far as working situations (lots of windows, nice view, interesting colleagues to talk with over lunch, etc.), getting there and back would involve about three hours of commuting total for me each day, plus buying a new car (we only have one), plus gas and maintenance costs, and also accident risks in icy winters here. That extra commuting cost made the job unfeasible for us financially, and the extra time cost would have been hard on my wife, who was still trying to finish her Working with Stories book while caring for and homeschooling our then-younger child. The extra time commuting would have precluded me doing any side jobs to help make ends meet. And of course there were no relocation benefits or such. I even found out later you also had to pay extra for parking there, which is probably to encourage commuting, but we were 45 minutes drive from a bus line going there.

I had applied for the job a couple of months earlier, and I had gone on a couple visits there including presenting at Capitol Camp 2011, but the wheels of NY State moved slowly. Likely I was for understandable reasons not a first choice given limited PHP and Drupal knowledge and also a decade of a spotty resume as a stay-at-home Dad with independent projects. When I had applied, we had maybe just enough cash/credit to maybe cover some of that extra commuting cost for a time, but months later, that cushion was gone to living expenses. So, I felt, for my family, I had to say “no”, after staying up much of the night making spreadsheets and trying to see if there was any possible way to cut our expenses further over the long term to make it at all feasible.

The financial constraints were not the entire issue though. Going in to talk with the group the next day (and not even mentioning the NBCU offer, knowing they could not match it, which may well have been a mistake), it seemed uncertain how much and how soon I could work on Rakontu-like ideas, public intelligence tools, and so on. So, even though my wife eventually said she was OK with me taking the job even if it did not make financial sense for our family despite the spreadsheets, it was also hard to justify a huge cost to my family while not even being certain I’d get to work on projects I’d hoped to do (issues not so much having to do with the technology team as the larger surrounding organizational context and related uncertainty). The NBCU contract alternative at least seemed sure to help my family financially in the short-term, whatever my own feelings or hopes. It seemed only fair to my wife to be able to support her finishing her free book, like she had supported me while I worked on my own free projects the previous years.

If they had offered me the job even just a week earlier, I would have likely taken it though. That was before I even applied for the NBCU contract, which had a really fast turnaround. I had put off applying for that NBCU contract for a week after I saw the opening, still waiting on the NY Senate job, but by then I felt it had taken so long it had probably fizzled, and so I had to apply elsewhere. It is much easier to justify taking a job you want to your spouse and yourself even if it can’t cover all the bills (and has other practical issues like fifteen hours extra a week in car) when you don’t have another ready alternative that would. Afterwards, I could have hoped for a financial miracle. :-) A miracle indeed might have appeared, as a few months later a company out-of-the-blue wanted to buy independent rights to our GPLd PlantStudio code and get some related consulting. We eventually turned that companies request down with me busy at NBCU and also us not keen on some other company suddenly owning a parallel copy of the PlantStudio codebase not under the GPL — a compromise we otherwise might have made had I taken that NY Senate job. One big advantage of working on-site for New York is they don’t make any claim on what you do off-site, unlike, say working as a regular employee developing proprietary software for IBM (although with the commute my ability to do other stuff would have been very limited). Also, a friend mentioned at one point months later that his elderly father living in a house around needed a caregiver/helper, so maybe I might have cut commuting costs by staying there during part of the week (but with its own family-time tradeoffs). So, one might say I just did not have enough faith things could have worked out if I had chosen the open source job despite the difficulties.

My whole time working for NBCUniversal that job decision and heartbreak of not doing open source software development at the NY Senate as part of a larger OpenGovernment/Drupal/CiviCRM FOSS community, weighed on me, being ever in the back of my mind. There was also my sorrow for having to disappoint that NY Senate group after all the work they put in getting me approved all the way up the chain there eventually — not to mention work they did checking my references and so on, and my the effort my references put in (Liam said he was “grilled” about deadlines) — compared to just a service bureau background check on me for the NBCU contract.

Still, with my taking the NBCU contract instead of the open source job, I knew my wife was getting the time she needed to finished an expanded version of her free Working With Stories book while also still homeschooling. And I was getting to spend more time with my kid than if I commuted three hours a day or lived somewhere else during weeknights. I don’t know about what car accidents never happened. And we were paying off debts and eventually piling up cash, which could provide time to do more free software development someday. And I was still working towards some free software here and there some evenings and weekends. And I was making my resume current again with full-time work for a big name in any case.

But, despite those benefits, I was not part of an open source community at NBCU the way I would have been working with the NY Senate then on something high profile like CiviCRM. And, whatever one can say about the dysfunctional Albany political situation and its prospects, helping New York government try to be more functional and consider all stakeholder perspectives (like by implementing ideas from my wife’s work) would hopefully have been better for the world than helping a liberal-leaning-but-in-practice-corporate-money-driven mainstream broadcaster stay live on the air.

But such was not to be. And what was most ironic about the situation was that working on so much open source software on my own time for free for so many years had effectively precluded me from doing so for pay back then. Otherwise, buying a second car or even a second small cheap home near low-cost-of-living Albany would not have been much of an issue. Driving in each day in a Tesla, listening to podcasts on technology or whatever, might even have been fun. :-) And we ended up (later) driving near Albany at least once a week anyway so my kid could participate in activities at a homeschool resource center (and had we lived around there, he could have done that more days per week). Our house in the Adirondack Park forest preserve was originally a camp-ish second home of a GE engineer who lived near Albany, and it had been our original fall-back plan when we bought it to keep it as a second home, like he had, if we had to move for work. But, Tesla fantasy aside, all that would have taken money we didn’t have from living on the financial edge for so long. Though, to be fair to FOSS, homeschooling was another expensive choice we made on top of making free and open source software (even as the two are connected in some ways relating to our interest in informal and decentralized education and decision making). If we had been willing to ditch homeschooling/unschooling, Cynthia could have easily made up any financial shortfalls from the NY State Senate job from increased consulting. But that was one sacrifice we were not willing to make — just so I could, essentially, take a job away from some likely equally qualified open source developer who otherwise might end up working for NBCUniversal instead of me.


On to NarraFirma (2014-present)

Why did you not use WordPress sooner?

While I’ve known about WordPress for a long time (almost since its beginning, as a long-time Slashdot reader), I have mostly maintained our own sites using plain HTML (generated sometimes by scripts from non-HTML source). This was mainly to reduce maintenance and security worries. Plus, of course, I’ve always had my own never-enough-time-to-realize ideas for content management based loosely on the ideas of using a triple store to implement a “social semantic desktop”. And of course, like many experienced software developers, I’ve long thought PHP was a problematical language, even as it continues to improve (although I could say the same about JavaScript).

While I had that one small beyondajoblessrecovery.org WordPress site around 2009-2010, I first started using WordPress significantly for a formal organization only about a year and a half ago. I set up a site hosted at WordPress.com for a local historical society (for which I am a trustee), migrating a static HTML site to WordPress. I chose WordPress after considering and setting up test sites in Drupal, Omeka, and Google Apps. I installed WordPress a few times locally, on shared hosting, and on Amazon Web Services to test it. I got WordPress to work in all of those places (even with getting outgoing email working on AWS via Amazon’s SES email services). However, I decided that hosting at WordPress.com was the most reliable option for the society. Most of the society’s board and membership has limited computer expertise (let alone knowledge about the details of computer security). The URL for that site is: http://edinburghistoricalsociety.org

What did you learn from deciding to use WordPress?

Going through the process of evaluating options for our historical society helped me to understand better how the web was changing. I began to see what a positive force WordPress was in providing a potentially decentralized alternative to using proprietary services like Facebook and Google Apps (even if, in practice, it is a good idea for many people to host at WordPress.com). Also, my wife and I began to see that her Rakontu project, and new ideas like our NarraFirma project, might be most easily installed and used as WordPress plugins.

I discussed NarraFirma at length above, so I won’t go into detail on it here.

Did you do any personal projects besides work on NarraFirma during that time?

During the time I was working on NarraFirma, I also goofed off a bit (relative to direct NarraFirma programming work), mostly improving my understanding of Node.js and what was possible using it. For example:

  • I spent a day or so writing TanksInYourBrowser with my son (who drew the bitmapped graphics). That program uses socket.io and Node.js to coordinate multiple tanks in your browser (in a very crude way). We were playing a lot of “World of Tanks” together at the time, and I wanted to show him how something like that might be implemented technically. I also wanted to give socket.io a first-hand test in hopes of eventually using it with NarraFirma somehow — even though I ended up sticking with polling to be easily compatible with WordPress-as-it-is-now.
  • I also wrote a small web-based Wiki-like information organizer as a Twirlip-related experiment with add-on IBIS support for Node.js, in part for my kid to organize information about tanks and such (not much interest beyond a page or two) and to use as a home wiki (again, not much interest so far). We run it on a Raspberry Pi. The IBIS add-on was created almost entirely within the environment itself.
  • I also wrote a command line version of the Pointrel system using Node.js which included a startable web server to browse stored triple-based content as HTML pages. That last system surprised me in being simple yet seemingly useful. It gave me ideas for simplifying the version of the Pointrel system NarraFirma is using, but I have not yet put all those ideas about further simplifications into practice in NarraFirma.

Twirlip Screenshot with IBIS addon

 


Next steps

So, what is next for you as far as paying the bills?

As I explained above, my wife and I took the profits from my NBCUniversal work and plowed them into the FOSS NarraFirma project over the past year or so. Now we’re running low on cash/credit again, and with NarraFirma finally out, I’m looking for paying work once again.

Ideally, rather than taking jobs making mostly proprietary software like at NBCUniversal or IBM, and then only later using that money to make free software, I’d like to combine my interests in FOSS and communications with a paying job doing mostly FOSS work. As I learned when evaluating options for my historical society, WordPress and Automattic support a substantial part of the web. It would be fun and purposeful to be part of supporting that existing user base. I would also like to help WordPress move forward in the new directions Matt Mullenweg has outlined (towards WordPress being a great application platform supporting complex JavaScript applications such as NarraFirma).

So, do you love PHP? Or do you love JavaScript?

As with anything, there are things one can complain about concerning PHP and WordPress (e.g. Ghost’s justifications for existence). It can be hard for a widely-experienced programmer to show much desire for doing a lot of PHP programming given alternatives — even if PHP development may be a necessary evil given the way web hosting culture developed. JavaScript likewise provides plenty to complain about, even if it can be used well, and even though transpiling to JavaScript from TypeScript and such provides other options for bigger projects.

Between those two widely-adopted necessary evils, which lie behind much of the web, I prefer JavaScript — even if I can still pine for Smalltalk, a language that underwent years of user testing and multiple versions before it became widely available.

Like anything, Smalltalk has its warts. One “wart” is simply lack of mindshare, which includes things like browser plugins, although there are some options like Amber that are based on compiling to JavaScript. Even Dan Ingalls, the original implementer of Smalltalk, started working in JavaScript for his Lively Kernel because JavaScript is easy to deploy. The best-written software in the world is useless if nobody will install it.

JavaScript has become the universal assembly language of the internet. And working in assembly language takes me back full circle to where I started programming, where you just build great things on top of whatever you have to work with, working within its limits as a creative challenge.

Why do you think WordPress is worth working on?

Practically speaking, it’s hard to beat the “one click” install process that WordPress makes possible for tens of millions of users. It’s also widely recognized that the WordPress community is amazing.

There is also the possibility of building JavaScript incrementally on top of a WordPress back end to take WordPress in all sorts of new directions.

What sort of new directions do you see WordPress or Automattic going in?

I may come to regret saying this :-) but I feel I would love to be part of the Automattic Janitorial staff on the receiving end of a steady stream of complaints/requests/suggestions by other Automatticians about things that are in the way of them doing their job as well as they would like (such as overwhelming streams of data like perhaps this document) with a mandate to write JavaScript-powered WordPress plugins to help make life at Automattic better and help everyone become more effective at doing good things wisely. For example, I’d really like a WordPress shortcode I could add to this start of a document to automatically create a graphical 2D concept map of most frequent terms with links to their usages as a visual index. (I did find a few pluging for making a regular table of contents though.) I would enjoy creating tools that many demanding and discerning people at Automattic want to use by preference without being forced to (otherwise, such tools would be failures in some way if people did not want to use them by preference over third-party alternatives or ad-hoc solutions). Automatticians would be a fantastic audience for really good tools. Working together, we could create some amazing new open source tools, some of which will likely increased demand for WordPress hosting.

I have faith those requests by Automatticians for new WordPress plugins and features for existing ones would overlap with a lot of software I would want to write for the world anyway.

Those new directions might be individual WordPress sites running tools like NarraFirma.

They might involve porting great ideas like Open University’s desktop-Java-based Compendium to JavaScript as a WordPress plugin.

Or, they might be shared systems that leverage perhaps-federated-wiki-inspired networks of WordPress sites to become a sort-of even better noosphere than what WordPress.com is now.

Pingbacks might be used in new ways to perhaps form an ever-growing organic tapestry of idea threads and running code. When I squint just right, I can kind-of see a network of WordPress-supplied-HTML-loading browsers running JavaScript as a “grand” network of interconnected Smalltalk-like message-passing VMs.

I can even see the possibility perhaps to build a triplestore into WordPress’s PHP core somehow (if proven useful through a plugin like NarraFirma first).

And there are other possibilities for WordPress becoming a more peer-to-peer application for messaging using data storage both on servers and via IndexedDB in the web browser.

WordPress has the social momentum to make any of those efforts more likely to succeed than if they were independent.

Dr. Heller: If you’re gonna bust into that mansion, you’re gonna need more than a station wagon.
Mr. Furious: At the auto yard, there’s this old Herkimer Battle Jitney. Sally’s always telling me to junk it!
Dr. Heller: Wait! You have a Herkimer Battle Jitney? That’s the finest nonlethal military vehicle ever made!

Can you be more specific about tools you’d like to build on top of WordPress?

There are probably great tools Automattic could use in-house which I would love to write — perhaps as part of the “Janitorial” team staff, making the lives of other Automatticians hopefully better. P2 and then O2 are great, but there are many new directions such tools could take, including informed by ideas from the educational and intelligence communities.

These ideas are embodied in tools like IBIS and Open University’s Compendium for helping think through priorities and organize ideas, SEAS-like collaborative decision-making tools and Angler-like multi-perspective tools for ensuring all voices are considered to overcome biases in making decisions, perhaps better outliners and issue trackers based on Automattician’s needs and wants, and so on. These are all tools that could make good JavaScript-powered WordPress plugins.

Years ago, at IBM, my wife collected ideas from the published literature and used them later to design Rakontu, NarraCat, and NarraFirma. However, there are so many other ideas already in the published literature that could be used to improve Automattic’s operations and the WordPress user experience if implemented on top of WordPress. Some of those ideas require going beyond “words” and into visualization tools though (whether visualizing 2D concept maps or visualizing 3D webs of structured arguments and so on).

As Scott Berkun wrote about the limits of P2: “Many conversations need to be visual.”

Other tools like better multi-user outliners might be just mostly text. Some tools like better whiteboards or spreadsheets might be mixtures of graphics and text.

If there was demand by Automatticians, it could be worthwhile to make JavaScript-powered WordPress plugins of some competitive or cooperative games or fun simulations like building virtual workspaces with 3D blocks or tending a virtual team garden (or even maybe tending a real remote garden together, including to show WordPress itself bridging the gap from democratic publishing to action in the physical world). Such plugins could support teams that wanted to get together virtually and chat while they play and relax and let their minds wander, a bit like what happens in real-life team meetups as Scott Berkun describes as happened when playing bar games, but maybe with more cooperative game options and with alcohol and travel being optional.

Some of these games might focus on increased understanding of stigmergic and chaordic processes central to the open source movement as well as Automattic’s culture.

Such new plugins might ultimately increase demand for hosting at WordPress.com if other companies adopted them as part of their own corporate training and team-building practices.

Building on the multi-user Pointrel technology I’ve developed for the NarraFirma plugin (or no-doubt an even better version of it, like with “PingPong” severs), as well as many other ideas, Automattic could eventually have the most amazing internal tools for keeping on top of emerging trends and immediate priorities as well as doing in-house training and team-spirit building. These tools could be all hosted as JavaScript-powered WordPress plugins. These plugins would be making the WordPress community stronger both by making Automattic stronger through fostering stigmergic and chaordic processes and also by being made available to WordPress users as plugins under the GPL so they can do the same for their own communities.

My father was ultimately a toolmaker working in metal; I enjoy being a toolmaker in digital media. Working with Automattic, I feel I could create some really useful tools for Automattic and the world.

Are there any other things you could help with at Automattic besides sensemaking and public intelligence tools?

There are no doubt a variety of plugins powered mostly by JavaScript that could be written on top of just regular WordPress as what might be called “admin themes“. But rather than existing WordPress admin themes which are mostly PHP-based or are mostly small tweaks via CSS, these newer admin themes could be implemented as JavaScript skins over just existing WordPress functionality so support task-focused WordPress use (a bit like NarraFirma being a complex web app with over 40 internal “screens”). With the new and improved core JSON support for regular WordPress APIs, there might be a whole host of admin theme skins for WordPress that ran as JavaScript single-page applications controlling the core WordPress engine via JSON requests.

So, there might be a “ghostly” :-) JavaScript-based admin theme plugin focused on just blogging which hid all the rest of WordPress’s enormous admin capabilities. In fact, there might be ten such admin theme plugins for “just blogging”, or a hundred, where users could easily switch between admin themes, same as it is easy to switch now between regular PHP-based site themes. Each such new admin theme plugin could be an experiment with what people liked, having more or less controls, and arranging controls and outputs in different ways.

This experimentation with new admin themes could all be done without putting at risk WordPress’s core functionality, which would always be available just by turning off the admin theme. Automattic would get user feedback from how such admin theme plugins were used as a way to decide where to go next with new and improved user interface skins, as in: “Let a hundred flowers bloom, let a hundred schools of thought contend.” (Hopefully the results would be better than when that phrase was perhaps insincerely coined.) Such admin themes might potentially reduce server load on WordPress.com servers, since as with NarraFirma, much computation would be happening in the user’s web browser based on cached data (like from keeping a list of all posts locally, with the list updated dynamically from a “PingPong” server or from polling). As another benefit, users who grow to like one specific JavaScript-powered admin theme on WordPress.com could stick with it, even as Automattic continued to innovate new user interface ideas in all sorts of ways for WordPress.com. Some of these new admin themes might be the work of lone coders, others might be the work of big teams. Automattic probably already sees all this potential, as otherwise Matt Mullenweg might not have been already saying for a long time that the JSON API is the future of a lot of WordPress development. I could help with this all somehow based on what I’ve learned working on NarraFirma and JavaScript/TypeScript.

Do you have any thoughts on whether WordPress might even move beyond PHP for the core?

PHP has been a big success as a language within a fairly limited web services scope for several reasons — including how easy it is to get started with a language whose default behavior is to just spit back out whatever you type in as the contents of a web page (a brilliant idea). But PHP remains a computationally inefficient language that is difficult to maintain large systems in — even if smart and dedicated people can do so, as Automattic’s history shows as does the history of several other web companies including Facebook.

PHP has seen widespread adoption for shared hosting with Apache and mod_php which embeds PHP inside of Apache so interpreter startup costs on page loads are minimal. However, with the rise of virtualization (e.g. $5/per month virtual private servers (VPS) at Digital Ocean), shared hosting’s days may be nearing an end. With virtual private servers, long-running processes that efficiently keep code and data in memory become possible, often resulting in reduced server load and quicker responses. With continually falling computing costs, it may soon just be easier for web hosts to manage all their accounts as VPS accounts instead of shared accounts, especially to reduce maintenance costs, security risks, and related complaints. “Ghost” is in part ahead of the curve on that trend as a blogging-focused Node.js app with emerging plugin support.

Cheap “wallwart” hardware (like for FreedomBox), the Raspberry Pi, Chromeboxes, and similar cheap computers (even just used mobile phones running Android) also push the envelope on what a personal “server” means. Efficiency tends to matter more at that low end, as does a certain coolness factor Node.js has for hobbyists following technology trends.

As Manuel De Landa points out in “Meshworks, Hierarchies, and Interfaces“, standardization at one level may create a stable interface that supports a large amount of diversity at another level. For example, standard electrical sockets with a standard voltage support a variety of electrical appliances. For another example, the TCP/IP standard supports a variety of web services.

In the long term (several years at least), more and more WordPress plugins (including admin skin plugins) will likely use the emerging standard WordPress JSON APIs more and more (especially given Matt Mullenweg’s previous promotion of the JSON API). Perhaps NarraFirma itself may help inspire a next generation of sophisticated JavaScript-centric WordPress plugins using a standard triplestore hosted in WordPress with a standard API. :-) As that happens, the WordPress community might eventually reach a point where it would be feasible for the WordPress core to rapidly transition to another implementation that supported the same JSON API interface as the current PHP-based WordPress backend does. (Automattic might reach that point sooner itself, given Automattic only supports a small subset of plugins at WordPress.com.)

That new backend might be implemented in TypeScript on Node.js (the most likely possibility, given the rise of JavaScript). Or, a new backend could be implemented in some other way (Java, D, Erlang, whatever). Or there even might be multiple backend implementations of the WordPress JSON API tailored for different purposes. WordPress, in that sense, might, like the Linux kernel to some extent, become more its API than a specific implementation.

But what about all the plugins written in PHP?

Of course, one may then ask, what about the plugins? As Matt Mullenweg has written, any good team could re-implement the core of WordPress in a year, but it is the plugin ecosystem that makes WordPress what it is. And most plugins (except something designed like NarraFirma) are tightly linked to substantial PHP implementations. However, as with most things, likely the 80/20 rule applies to plugins, and only some small subset of the plugins are really important to the community, the rest being essentially small experiments or “me too” overlapping duplicates. Likely many of the most important plugins will migrate over the years towards using the WordPress JSON API on their own and so will become more JavaScript-centric and have smaller and smaller PHP backends (somewhat like NarraFirma). The plugins that don’t move that way on their own could be migrated by hand to JavaScript/TypeScript or whatever, perhaps using a semi-automated process. Such a plugin conversion helper tool might be created by building on php.js to make the initial transition easier, aided by a translator that does a lot of the heavy lifting of converting PHP to, say, TypeScript (like when I wrote tools to help translate our Delphi apps to Java and Python, and a tool before that to help convert our C++ to Delphi).

Transitioning WordPress and all the WordPress plugins to a new language like TypeScript (or whatever) would no doubt required an enormous community effort full of complex politics and winners and losers and much strong feelings. There are many experienced PHP developers out there who might deeply resent having to shift to coding in JavaScript/TypeScript. That will be true even if more and more web developers are learning JavaScript/TypeScript and transpiled-to-JavaScript languages like CoffeeScript to do more sophisticated web front ends. Strong complaints about moving away from PHP will be guaranteed even if the overall web developer mindset itself continues to shift quickly towards doing more in JavaScript (or languages that transpile easily to JavaScript like TypeScript, especially after seeing projects like Angular 2 adopt TypeScript). Such a transition would not be something to be done lightly. It would be painful. It would produce a lot of noise. If managed badly, such a transition might spell the end of WordPress and even Automattic. Even just discussing such a transition carelessly may seem dangerous if it creates uncertainty and hesitance among plugin developers or WordPress users. And such a transition remains years in the future.

Still, it seems to me that such a transition is likely inevitable given ongoing web trends, including towards JavaScript everywhere (for good or bad). If so, ultimately the issue is whether Automattic will lead such a transition as an act of “creative destruction” or if the momentum might eventually go to Ghost or other similar attempts to move beyond PHP?

[the team are working on the Herkimer Battle Jitney to prepare it for battle]
Dr. Heller: It’s a process which results in an accelerated flow of electrons that creates such a powerful magnetic force.
Invisible Boy: So this is, basically, like a huge electromagnet.
Dr. Heller: Well, actually, it’s an electro-nuclear-magnet. It’s the next inevitable phase.

Still, while I outline this idea of the WordPress core transitioning to JavaScript for completeness and to show I’ve thought at least a little about challenges Automattic might face down the road, for me and what I want to do at Automattic, it really doesn’t matter. WordPress in PHP is good enough for me to build complex multi-user apps on, as NarraFirma shows.

Replacing the WordPress core might make it faster or easier to maintain, but those are not big limits for me at the moment.

Such a change would just let existing servers handle maybe a (probably) six times greater load under some conditions. It might mean half the WordPress engineers could eventually do other things if WordPress core code got easier to maintain or extend in TypeScript instead of PHP. Still however nice that might sound in theory, we all know practice might fail to mean expectations, and the whole venture might be too risky to even contemplate, even if Ghost essentially is forging ahead in that direction (but without WordPress API compatibility).

As far as I’m concerned for the projects I outlined above, the WordPress core could never change from PHP for a hundred years and I could still write grand things on top of it. These computational inefficiencies just mean servers cost probably six times more than they “should” (if my estimate is correct). However, the benefit of not changing from PHP is that existing people in an existing community can keep using everything they have learned about PHP and so can focus their learning and development efforts in other directions without any speed bumps of some controversial conversion. And that itself is a big benefit of the status quo, and not to be easily discounted.

So, I’m not pushing this change even as I outline it as a possibility. What else might you expect from a JavaScript engineer but to see the potential for JavaScript everywhere, maybe even places it should not be for whatever reason? :-)

Probably in a few years one can see where the WordPress community including plugin developers is focusing their energies and re-assess the situation.

What about a proxy?

There are of course many other long-term possibilities involving PHP or other server-based code. I could imagine creating a WordPress plugin that could make a WordPress site serve as a web browser proxy for archiving, filtering, and adding new JavaScript-base tools to any visited web page for crowdsourced public intelligence. That is as opposed to web browser plugins that do something like that but aren’t running on a highly connected server which could quickly talk to an Akismet-like server for crowdsourced rating and sharing services. Such a proxy connected to over ssh might even be useful for people living in undemocratic countries who have a WordPress site elsewhere. But that’s all a huge stretch.

Still, such a proxy could also potentially do things like remind people about (or enforce) personally-chosen limits on web surfing time at various websites (or parentally-chosen limits, for younger kids). Or such a proxy could provide people with good statistics to monitor their own web surfing behavior and track their own changing interests over time, where such metrics might eventually lead to a more efficient use of a person’s time and maybe even to improved overall health and overall effectiveness. So, the upsides might be worth the difficulties.

Of course, such a proxy could run stand-alone if it was a good idea, as could many of the other possibilities I mention. But WordPress makes software easy to find, rate, install, configure, and upgrade, and that may be the difference between people using some software or not.

What about text mining for Happiness tickets?

Another server-based change could be to support Happiness Engineers using text mining on previous closed support tickets. When I read Scott Berkun’s account of his initial three weeks learning to handle Happiness tickets, it seemed to me that if I was doing that, I’d want to review previously closed tickets to learn both what to say in response to common questions and how to say it. While that would be educational, with probably on the order of a million closed tickets, where would I start (other than at random)? Then it occurred to me there could be some search function for tickets (and no doubt there is). Then I though about my wife’s early work on narrative at IBM Research, where she used IBM’s Intelligent Miner for Text to do clustering.

What if there was some service at Automattic for happiness engineers, so that when they looked at a Happiness ticket, there was also a list of similar tickets from the past available based on big data mining and clustering done somehow based on some ticket similarity measures? If such a service proved useful to Automatticians, it could also eventually perhaps be rolled out to the general WordPress community for self-service. Granted, most of the people who open tickets probably still would not use that, but such a service might still help consultants and community members help others. I suppose access to that service could even be charged for perhaps. Still, there might be privacy issues involved in making such a service available outside of Automattic, so that might not be feasible. In that case, text clustering could still be used to help Automattic figure out what should be in a support FAQ and to determine where future improvements to WordPress should be focused.

What about supporting user feedback on emotions?

Another idea came out of reading Scott’s account, and it is also inspired by Team Social’s efforts to give feedback for new posts and comments. What about adding support for providing user feedback for closed Happiness support tickets? This could be done by including a web link in a final email to close a ticket.

When I, as a service user, open a support ticket at some service, and the ticket is answered to my satisfaction and closed, I always am not sure whether to send a thank you response. I want to say thank you, but I don’t want to create confusion by posting more on the support topic. But not saying “thank you” seems rude and also possibly demotivating to the person who helped me. It would be nice if there was some way to respond at that point in a way that clearly was not a request for more support — like perhaps just using a slider to give some number of applauding hearts (or maybe booing raspberries too, just for contrast forcing users to make a choice about how they feel). Such out-of-band feedback could then be clearly distinguished from more questions about the topic.

The same approach could perhaps also be worked into the WordPress plugin submission process, where the ultimate acceptance email confirming SVN repository access says specifically not to send a thank you email as it just makes more work for the people involved.

Sometimes companies do customer surveys after support interactions, but it is often too late then. And in any case, who (except someone unhappy) wants to fill in a long survey later when they really just want to say thank you (or not) at the moment of closure on the specific issue?

It might also be useful to ask users to indicate their own mood when they first make a support request, and to indicate their mood after the request is fulfilled, where the difference in mood, as opposed to the absolute final mood, would be a useful indicator. This could add some extra communications “oxygen” related to emotions that might be otherwise apparent when support was done face to face or over the phone.

Hopefully, these extra emotional indicators would help Happiness Engineers to stay in a positively-reinforced flow of helping people, although obviously there are other dimensions like difficulty of request or difficulty of requestor.

Happiness engineers would of course perhaps sometimes to get negative emotional feedback if raspberries were an option, and they then might have to decide if the feedback applied either to the requester just being like that — perhaps obvious by past ticket responses — or whether it was from the interaction having gone poorly for another reason, which they could perhaps learn from.

This general idea might also be applicable with P2/O2 discussions as well.

These ideas for improve happiness feedback build a bit on my wife’s concept for NarraFirma and previous work, where quantitative metadata (often about emotion) is collected about qualitative experiences (generally described in textual prose), as a form of “mixed methods” research. Some of her other ideas from Rakontu (community software for sharing stories with metadata) might also be useful to consider for the WordPress code or plugins.

Moving from SVN to git?

It could be worthwhile to help Automattic transition the WordPress plugin source code control from SVN to git, the same as Melissa Anderson did for Drupal. I have a lot of git experience, including from helping out team at NBCUniversal transition from CVS to git.

What about a virtual WordPress user?

Building on my past experiences with writing tools for regression testing, it might also be great to implement a JavaScript-powered “Virtual WordPress User” who, within a virtualized test environment, was continually setting up WordPress blogs, making posts on them, commenting on them, adding and removing plugins, and so on. Such a user could report differences based on comparisons with past WordPress behavior. This could help alert the Automattic team to unexpected changes to the WordPress user interface experience as side-effects of code checkins. As with my work on XSL-FO testing, if the changes were to be expected, the virtual user could be told to change the expectations as to the behavior.

What about improved proofreading and writing assistant tools?

It might also be interesting to rewrite Automattic’s acquired “After The Deadline” grammar checker in JavaScript. There are many brilliant ideas in it that could benefit by being brought to an even larger audience. If rewritten in JavaScript, either directly or building a JavaScript-based interpreter or transpiler for its specialized code, that code could be installed more easily using Node.js or parts of if could run in the web browser (or maybe all or it, with some redesign based on the 80/20 rule to reduce training data size). Being in JavaScript might also to make that codebase more maintainable given how a lot of it is in an innovative but not-broadly-known “Sleep” custom language.

The “After The Deadline” idea might even be taken a bit further somehow (perhaps called “Before the Deadline”? :-) into a general tool to help novice writers learn to write better. When I was in college, editing my papers under VMUTS (Unix for IBM’s VM), there was available a great tool for spell checking, grammar, and style (likely “Writer’s Workbench“). I still miss it. Following most of its suggestions not only greatly improved my specific term papers back then, but also, in the process of using it, that tool helped me become a better writer overall. But my use of such a tool also built on many year of patient writing instruction from a variety of teachers covering many other aspects of writing (even poetry). It is these other aspects of writing (outline, tone, audience, purpose, metaphor, ruthless cutting, etc.) that an improved tool or toolset might help with through online tutoring. Maybe one “life-line” tool in that toolset could even provide a for-fee (or volunteer) service through a WordPress plugin connecting really good writers and editors with people who wanted to honk for their services? Maybe Automattic could get a cut of such transactions somehow?

I could certainly use such a toolset to continue to improve my own writing — as anyone sticking with this so far has no doubt realized. :-) (The “After The Deadline” service seems to time out on a document this long, BTW.) Using such tools is as opposed to taking up my wife’s time, who proofread earlier shorter versions of this document — before mostly giving up as my resume transformed more into a 120 page 59000+ word manifesto of what I’d like to work on at Automattic. :-) A manifesto which has strained the limits of what you can do in one WordPress page in various ways, as the editor gets slower under FireFox (unusably in Text mode but not Visual mode, strangely), and even our web server starts throwing up random FastCGI errors when opening the edit page or saving the document also probably from timeouts. So don’t blame my wife for the various grammatical errors and typos no doubt still in here — or its lengthy rambling structure which she advises against.

Shoveler: (examining the strange device on the tripod) And what do you call this?
Heller: Careful, careful, that’s a Blame Thrower.
Shoveler: A “Blame Thrower”? (accidentally activates it) Oh, oh, I’m sorry,
Spleen: (caught in the beam, his voice goes high-pitched and distorted. He whirls on Bowler) It’s your bad attitude that’s hurting this team!
Bowler: (also in the beam, her voice does the same thing) Stop spitting on me! I took a shower already today!
Spleen: Well you need another shower, ’cause you stink!
Bowler: Want me to get that zit for you?
Spleen: Ooh, I’m really shaking! Ooh, I’m really scared!
Shoveler: Oh… (he switches the thing off)
Spleen: (back to normal) I’m sorry! I’m sorry if I spit all the time! I’m sorry.
Bowler: (back to normal) I’m sorry. (Spleen goes to give her a hug and she fends him off) No no, no. No no no. No.
Shoveler: Doctor…you /are/ a genius.
Heller: That’s what the card says.

Sorry I could not do better, even after over 256+ revisions. Maybe someday I will — perhaps helped by WordPress plugins for summarizing and organizing?

This document might also be helped by moving beyond the self-imposed constraint of working within one WordPress page. That was constraint initially mostly just so I could justify attaching a shorter version of it to my application email (I like distributed data for backup and redundancy). But as an unguessable page link on a WordPress site with the Vantage Premium theme, it also provided some faux temporary privacy of an otherwise uncrawlable link under WordPress while I continued to work on it — while meanwhile Automatticians (or maybe other people I might send the link to) could potentially still access the latest version without a password. While this document could become something much better (I hope) by transforming content from it in new ways (whether splitting it up into pages on a multi-page WordPress site, or putting the stories in it into NarraFirma with metadata, or some other creative reformulation), it is starting to feel to me about done for what it is as (by now) a small book.

Whatever the merits of the content I’ve written, it’s kind of amazing to think you can essentially write an entire small book these days while working within the constraint of a single WordPress page. That’s a real tribute to WordPress technology and a modern FireFox web browser. :-) Although I did cheat a little near the end by cutting and pasting the HTML text a couple of times into another editor with a grammar checker and then pasting the improved results back in to WordPress — but of course, as above, that’s one of the things maybe I could help improve when working at Automattic. :-) And that use of automation was mainly to spare asking my wife to look over it yet one more time, as she proofreads much better than any program I know of (synesthesia helps) — even when she feels the whole document being proofread is a dubious enterprise at best as far as being a good way to get a job anywhere.

Do you expect to do all these things from day one at Automattic?

So, lots of things are possible at Automattic starting from a base of WordPress at different levels of effort and opportunity. And of course, other learning experiments in the spirit of the new Selfies app are possible by building complements to WordPress without the new code necessarily being a direct part of the WordPress product itself.

As Scott Berkun wrote of his time at Automattic, code and creativity are not the limitations at Automattic with a huge backlog of todo items; clarity (or “priorities”) is the issue. I may even have perhaps added some to the backlog by my suggestions here. :-) Although no doubt Automatticians have already thought of many of these ideas if these suggestions are any good. And possibly some of them may even have been implemented already, but I am just not aware of them.

Obviously, setting the priorities for such things involves a complex social process at Automattic, based on assessment of the ultimate user experience. So, I’m not saying I have to do any of these specific things right away, or even that they are all for sure worth doing. I’m not even saying I could do all of these things in any reasonable time frame (as, all together, they are probably dozens if not hundreds of person years of work). I’m mostly trying to demonstrate some engagement with the kind of opportunities and concerns Automattic has likely already been considering for years, but without quite enough people power to address them all.

What other challenges do you see when working on such ideas at Automattic?

There is also though the challenging balance of doing in-house development at Automattic of new plugins versus stepping on independent WordPress plugin developer toes. Millions of people send money regularly to Automattic because WordPress is a good thing in their lives — but also perhaps partially in hope to supporting further development like of new JavaScript tools as plugins as above to make WordPress even better. However, at the same time, many independent WordPress plugin and theme developers have created business models related to selling access to WordPress plugins, themes, and related support. That’s a challenging slackline to walk when pursuing some of my suggestions above, although that challenge might be approachable by developing any new plugins on GitHub so developers could easily fork them to build on them in new directions. Also, a “PingPong” server enabling low-latency near-real-time web multi-user interactions on top of WordPress could make possible all sorts of new opportunities for third-party developers to enhance existing products or create new ones.

As a risk to be managed, a few of the things that might be nice to write as new tools might ideally support allowing site owners to add custom JavaScript. Custom JavaScript is not allowed on WordPress.com for good reasons to reduce cross-site scripting (XSS) concerns. It is not clear how to address that XSS concern in a WordPress.com context — although maybe together we could find a way to manage that risk somehow?

So, these are all just more things to consider of many good possibilities already out there on P2/O2 lists somewhere. No doubt these possibilities all might have their hurdles and challenges to think through about whether they made sense to be done at Automattic at the moment.

Besides being paid to work on specific projects as above, why do you think you would like to work at Automattic?

Automattic clearly understands what Dan Pink says about regarding motivation through autonomy, mastery, and purpose.

Automattic also embraces stigmergic and chaordic processes internally as an implicit management idea (even if I’ve heard little formal talk about those two as secrets to Automattic’s success). It would be cool to work in a company that embraces such ideas, whether formally or informally. I provide some more information on those two because I want to be sure Automatticians are aware of the ideas and the existence of a related literature, and I hope it is fairly obvious how they connect to Automattic’s policies and aspirations.

What is stigmergy?

From the first link on stigmergic collaboration:

The following represents some of the current findings of the author’s PhD research on and around collaboration and stigmergic collaboration, and comprises the core components of the theoretical framework guiding this article:

  • Collaboration is dependent upon communication, and communication is a network phenomenon.
  • Collaboration is inherently composed of two primary components, without either of which collaboration cannot take place: social negotiation and creative output.
  • Collaboration in small groups (roughly 2-25) relies upon social negotiation to evolve and guide its process and creative output.
  • Collaboration in large groups (roughly 25-n) is enabled by stigmergy.

What are chaordic processes?

And from the second link on chaordic processes:

Join a Purpose-Centered and Principle-Based Community

The Commons is purpose-centered and principle-based. The common purpose that all owning members share is:

To develop, disseminate and implement new concepts of organization that result in more equitable sharing of power and wealth, improved health, and greater compatibility with the human spirit and biosphere.

Owning members observe the following principles when participating in activities of the Commons:

Principles of Practice

  • Work to ensure that all people, by right of birth, have adequate necessities of life, including clean air, water, food and shelter; an equitable share of wealth and resources; and opportunity to develop their full physical, mental and spiritual potential.
  • Work to ensure that human capacities, technologies and organizations protect and support, not systemically alter, degrade or destroy, the Earth, its diversity of life and its life support systems.
  • Work to ensure interdependent health and diversity of individuals, communities, institutions and cultures.
  • Resolve conflict creatively and cooperatively without resort to physical, economic, psychological, social or ecological violence.
  • Freely and fully exchange information relevant to the Purpose and Principles unless it violates confidentiality or materially diminishes competitive position.

Principles of Organization

  • Be open to owning membership by any Individual or Institution subscribing to the Purpose and Principles in conducting activities of the Chaordic Commons and Terra Civitas Initiative.
  • Have the right to self-organize at any time, on any scale, in any form or around any activity consistent with the Purpose and Principles.
  • Conduct deliberations and make decisions by bodies and methods that reasonably represent all relevant and affected parties and are dominated by none.
  • Vest authority, perform functions and use resources in the smallest or most local part that includes all relevant and affected parties.
  • Educe rather than compel behavior to the maximum possible degree.

The principles are the “organizational DNA” found in each and every “cell” of the Commons, no matter how it grows and evolves. Participation in the Commons provides owning members an opportunity for experiential learning about new ways to organize in a self-organizing, self-governing, self-evolving community!

How do these practices relate to Automattic’s move to teams?

In “The Year Without Pants”, Scott Berkun writes:

“One of my recommendations was to move from a flat organization to a team structure. This was an obvious solution to a major frustration among employees: how centered thing were on Matt and how they wanted to take on bigger projects than they could alone. But the challenge was that the culture strongly resisted hierarchy.”

While I’m sure he meant well, I’m hopeful that by now people at Automattic realize (especially as Automattic continues to grow) that there are other ways to address that challenge than just teams (as good as small formal teams can be).

Stigmergic and chaordic processes in particular provide alternative ways to find the right balance of meshwork and hierarchy for every challenge. As Manuel De Landa writes:

“Indeed, one must resist the temptation to make hierarchies into villains and meshworks into heroes, not only because, as I said, they are constantly turning into one another, but because in real life we find only mixtures and hybrids, and the properties of these cannot be established through theory alone but demand concrete experimentation.”

Automattic seems well-positioned to make the most of those stigmergic and chaordic possibilities as it seeks a healthy mix of meshworks and hierarchies that best supports the WordPress community and the larger web around it.

As Scott Berkun also wrote:

“After years of leading projects, the best thing I’ve learned is that I have to periodically shift between thinking small (bazaar) and thinking long term (cathedral).”

Ignoring the fact that the global bazaar is much larger and long-term than any specific cathedral, this shows him almost making the points made by Manuel De Landa (meshworks, hierarchies, and interfaces), Dee Hock (chaordic), and Mark Elliot (stigmergic), but without using that specific terminology or linking to a larger literature (beyond incidentally referencing the book Emergence much later). As is said in the Google video on HR and statistics Matt Mullenweg linked to from his site, maybe companies should strive to get more ideas from academics (and then rigorously test them in-house, as Google did)?

Automattic and recovering from the not-quite-Halloween porcupine disaster?

After I had written this essay (short book by now it seems), I was out walking our dog Brownie (my kid picked the name), an almost-three-year-old beagle-terrier-other-things mix, on the main road that goes by our house, enjoying the colorful oranges and reds of the autumn leaves two days before Halloween. I had on my usual dog walking apparatus on top of my regular outdoor clothes — an add-on belt with a training bag with treats and dog waste bags (and a clicker I never used). I held the handle of a retractable long lead. As a backup, that handle was also connected via another short belt to the belt around my waist just in case. It had rained heavily the day before and we had not walked Brownie (one of the few days in her life she’s ever missed a walk), so I was taking her on an extra long walk. At one point early in the walk, I heard some screeching behind us, which I put down to coyotes, although it had sounded a bit different from coyotes, who I also heard more for sure up ahead, and it seemed odd to have two packs of coyotes near each other.

I don’t usually walk this far on our walks. We had walked by one large homestead which has a construction/repair business and had Halloween decorations out that had been left out continuously for years. One decoration was a Jason-looking hockey mask on one tree (next to some friendly-looking gnomes along a rock wall). Another decoration was a stuffed effigy of a person holding a pitchfork hanging from another tree. That was all typical Halloween humor. Of course, I was not sure about the point of leaving such decorations up continuously for years, and that was a little odd, but rural life with endless things to do is sometimes like that. And maybe they had some humorous-or-not point to make.

While that place was a bit odd, the next house a ways past that one was a place I really did not like to go by. There had been a marital murder/suicide there a few years ago. It involved the son and daughter-in-law of someone I knew from our historical society. It was a a terrible tragedy leaving behind two kids to be cared for by their grandparents. I mention that tragedy in this Open Letter to the Intelligence Advanced Programs Research Agency (IARPA) as an example of what better sensemaking tools might help reduce in the sense of making communities better and happier and healthier places to live. That tragedy made me think more if there was anyway the web could help prevent such things by helping unite local communities, especially my community.

As is said by Leslie H. Farber in a book titled “Lying, Despair, Jealousy, Envy, Sex, Suicide, Drugs, and the Good Life”, a completed suicide is a bit like someone falling suddenly thorough a trap door. We can work backwards from such an event, and we can think of all the things we might have done to prevent the person from stepping over the trap door and falling through it (like were we kind enough or stern enough, did we give too little advice on nutrition or too much, and so on). But, ultimately, falling through a trap door could in theory happen to anyone. There are people you think have miserable lives who just keep truckin’ on, and others who you think have everything who don’t.

We can at best just reduce the likelihood of such terrible events by reducing distress in our community, as well as put other policies and processes in place in our organizations as at Automattic. We can create more opportunities for “eustress” (good stress that ultimately makes us healthier, as an aspect of “positive psychology“). We can even encourage “post traumatic growth” as a response to disasters in our lives and choices we make about how to respond to them, and hopefully move past resilience into thriving.

This essay might have been very different if I had dwelled mainly on issues my sisters and my mother struggled with (starting with frequent migraines not then understood were coming from things like sulfites which could be mostly avoided or reduced by magnesium supplements and such), rather than emphasizing the better parts of my father’s life and career accomplished through persistent hard work and continuing education despite various challenges. However, all of them demonstrated resilience and thriving in different areas of their lives in their own way (along with sometimes not), as do most people (as do I). We all make choices, including what to emphasize in our writing. And we all generally can learn and continue to improve our choices.

Beyond individual choices, that likelihood of distress leading to tragedy is also affected by how communities and organizations choose to their time and money (or choose to let others spend it supposedly on their behalf). For example, in my community of about 1500 year-round residents (and many more seasonal ones), the Iraq war has cost about US$10,000 per person, or about fifteen million dollars (on top of an ongoing defense cost of about US$3000 per year per person). I’m all for spending a lot of money on security and defense — even perhaps more money than we do now, like to combat existential threats from supervolcanoes, asteroids, Singularities, plagues, earthquakes, water shortages, global climate change, systemic economic/social collapse, arms races, and more. But, I am only for spending the same or more money on defense if we are securing the right things and defending ourselves in effective ways and not otherwise just not wasting the money on obsolete models of national security. That fifteen million dollars could have, say, been used instead to build an amazing local community center for our small town with a library, an indoor track, an indoor pool (we get a lot of snow here), a public learning center, a makerspace, and so on. And/or it could have funded many more local events to forge personal ties than the two such events our small historical society runs funded via bake sales and such. And/or it could have funded better social services for local families in crisis (especially with the continuing death grip of the Great Recession these past few years despite whatever some well-paid economists say). All those investments in true “homeland security” would have made my town of Edinburg much stronger and able to cope with whatever threats might arrive, from the occasional ice storms to a major forest fire and even all the way to the very remote possibility of foreign invaders from Wing IV scheming to destroy our way of life.

Like good nutrition, making our town healthier in those ways would have made it more resistant to all kinds of specific diseases that otherwise have to be addresses individually at great cost. That money could have secured and defended my town from many tragedies — even if some tragedies will always slip by our defenses no matter what we do, just as some diseases will get past even the healthiest immune system and then will need specific cures.

Instead of building health here in Edinburg, that money was used to, among other things, seed a foreign country (and US service people) with radioactive waste which will likely cause generations of suffering through birth defects. That is such a horrible outcome of how our tax (or national debt) money was spend that most US Americans can not even face the abstraction, let alone look at the actual pictures of the results. Could a better web have helped the US government and citizenry choose to spend that money better and in a healthier way? I don’t know for sure — but I can hope. When you are a computer software developer with a JavaScript hammer, every problem looks like a webapp. :-) Even as you know realistically, by itself, that is not enough.

I could not help but think how much better my mother’s last few years might have been if that Iraq war cost had not been translated into nursing home staffing cutbacks via Medicaid, where my mom, who had already paid in personal suffering for WWII, had again paid the price for a new war in personal suffering and indignities. And I could not help but think that this family of another historical society board member might still be intact instead of in part under tombstones if that fifteen million dollars had been spent differently somehow. But no accounting of Iraq war deaths is going to include the people up the road from us who are both now dead. And, indeed the connection is indirect and speculative. The tragedy might have happened despite what anybody did. It’s just how I felt about all that, reflecting on the memory of the one time years ago I had waved to the wife and said hello, while I was walking a different dog for a long walk as she was out waiting for the school bus to bring her kids home from school. I don’t know specifically what we all could have done differently, but it was sad, and as much as that family had obviously failed itself (as many families do in lesser ways), our community had no doubt failed them too, as an interwoven set of cause-and-effect with no clear beginning or end. And, being part of that local community, I was part of that failure too. (Technically, that house is actually over the town border line a bit, but they still felt local to our community.)

So, for me, that was a spooky house to walk by, especially near Halloween. I had to wonder about people who would buy a house with such a tragic history even at a big discount. The new owners also had an unfortunate experience last year, driving through a guard rail (which collapsed) over a small bridge at a sharp turn near where we live, with their pickup truck ending up in the creek.

As I walked by that house (set way back from the road behind trees), I heard a male voice there shouting loudly in frustration at his dogs to be quiet (which I knew was not going to help much, including because dogs may interpret shouting at them as if you are barking along with them at the same threat). The dogs were likely barking because of the coyote noises. The coyote howls would set Brownie off too when she was at home, and indeed that can be frustrating to deal with — even as I know Brownie is just trying to keep my family safe us with a bravery beyond her size — as part of the five doggie D’s of home security (deterrence, detection, denotation, delay, and defense). But walking Brownie by that house may have set off those dogs too, another reason to hurry by to perhaps reduce the problem.

As we finally passed by that house on the return trip, I also thought about a book, A Wizard of Earthsea. In one chapter, the wizardly hero is in a location with a large stone somehow connected to one of the “Old Powers” of the Earth, which are maybe not quite “evil” but are certainly different and evil-enough-seeming in practice. The wizard knows that if he touches the stone that the Old Power will own him forever and he will have to do its bidding with all his wizardly powers — something like an expanded version of computer contracting I guess. :-) I wondered if there was some evil or bad luck about the home’s location, like some “Old Power” stone.

With these and other related thoughts in my mind about disasters and how to avoid, prevent, or deal with them, Brownie and I walked on.

We had been walking for about an hour or so. As we had turned around at some point, we were now about twenty minutes from home. We were getting near to an old barn by the side of the road, probably back around where I had heard the original screeching that I had guessed was coyotes. It was starting to get dark.

It’s easier to let your guard down on the way back home on the same road, which can have some beneficial affects for creativity. My mind had started wandering away from the walk itself and into thinking about Automattic, and I was paying less attention to our dog. I had been fancifully thinking about things I might ask Matt Mullenweg or other Automatticians were I ever lucky enough to work at Automattic, musing over the witty conversations Scott Berkun recorded in his book about his first days at Automattic.

Next thing I knew, Brownie zoomed across the road, jerking the retractible leash out of my hand where I loosely held it, which in turn yanked the extra small backup belt attached to the treat bag belt around my waist, which then popped that other belt open (given it just had a plastic snap buckle), and ultimately left the training bag and belt in the road as Brownie continued to zoom ahead. She moved beyond the edge of the other side of the road and into a clearing in the brush, where she poked her nose at a brownish bushy-looking beach-ball-sized something before I could grab the retractible lead from the road to pull her back. As with many accidents in the woods, the common refrain is: “It all happened so fast!”

I used the retractible leash to pull Brownie back as quickly as I could, but it was too late. I saw about twenty porcupine quills sticking out of her muzzle and nose, probably there from a swipe of the porcupine’s tail. Even as I would often get new ideas or solve problems when walking dogs over the years, my dog was paying a heavy price for my inattention to our walk together that time. Sorry, Brownie. :-(

What to do? In all my years of dog walking, I had never had this happen before. In retrospect, I should have been more careful around the old barn because we had seen porcupine quills on the ground there before. My wife had even collected some from there out of curiosity. Nor had I ever read much about porcupine quills or related first aid. Again in retrospect, that was a foolish omission after seeing the quills on the ground in the past. Also, I hadn’t realized it then, but around Halloween is porcupine mating season, and it is possible the earlier shrieking I heard was not coyotes but a porcupine and I had attributed it incorrectly. It was near nightfall, and porcupines are mostly nocturnal. In any case, I should have remembered to be more careful when going back by the noise. So, I had missed multiple warning signs. My backup system had failed with the treat bag belt buckle popping open when tugged on. The treat bag and belt was not a very expensive item and I had been using it for years, probably with the buckle’s plastic fatiguing all that time. I much later noticed the plastic buckle had a crack, either from that big yank or from previous ones where it had popped open before — so I could have had a better belt buckle for the training bag. Instead of the clicker I never used, I could have had a pair of pliers or a first aid kit with forceps or at least a rubber glove. However, 20/20 hindsight is always perfect vision. And of course, long ago, we could have gotten a different mix of dog with less hound or terrier (breeds that want to either chase things or kill rodents, even super-sized rodents like porcupines).

While I always carry my cell phone when I go walking, cell phones don’t work reliably around where we live, so that option was not available at the specific location to call for advice or asking for help either. Our town supervisor had been trying to get better cell phone coverage for years for safety reasons (like when someone slips on ice), but the major carriers apparently did not feel our town would be profitable enough. It did not help that seasonal residents and visitors generally liked the lack of good cell phone service as a forced way to get away from their electronic lives for a while. I had thought years ago of buying a satellite phone, but I never had, given the operating costs are expensive and they are bulky. We had family radio walkie talkies, but we did not normally carry them, and in any case, they are short range and also useless unless someone is listening at the other end.

Essentially, I was on my own in the middle of an accident — without much of a clue about what to do about it, given inadequate tooling, training, and networking.

One of the most insightful things I’ve learned was from a short Driver’s Education film I saw as part of a class I had to take in New York State to get a drivers license. It was called “The Seven Factors That Make An Accident”. It argued there were (typically) seven factors that go into a car accident, things like bald tires, slippery roads, inattentive drivers, poorly designed intersections, people hurrying, stuff in cars obscuring vision, sudden braking, and so on. The point of the film was that if you could reduce even just one of those factors, like paying more attention to the road or making sure your tires were in good shape (or maybe lobbying for better designed roadways), you might prevent the accident which required all those factors to come together.

If I had been paying more attention at the time, or had had a stronger training belt buckle, or Brownie had been a different dog, or I had walked at an earlier time, the accident would not have happened. Or if I had thought through the implications of the Porcupine quills by the barn and done related study, at least I would have know what to do when it did or could have prepared my walking kit better to respond. But I had done none of that. And now Brownie was in pain.

A more general equivalent idea to the “Seven Factors” one is “The Swiss Cheese Model of accident causation“. As Wikipedia says, that model “likens human systems to multiple slices of swiss cheese, stacked side by side, in which the risk of a threat becoming a reality is mitigated by the differing layers and types of defenses which are ‘layered’ behind each other. Therefore in theory, lapses and weaknesses in one defense do not allow a risk to materialize, since other defenses also exist, to prevent a single point of weakness.”

A similar model more specific to resilient fault-tolerant computer systems that I found while studying this issue contracting at NBCUniversal is “A Conceptual Framework for System Fault Tolerance“. That paper talks about different ways to reduce and manage the risks of faults in computing systems so that that faults do not propagate into visible failures at the system’s boundary. That paper discusses how to avoid, remove, and tolerate faults by various methods which, taken as a whole, detect, diagnose, contain, mask, compensate for, and repair faults. That paper provides a consistent terminology for discussing the challenge, a terminology well worth considering.

All three models essentially say the same thing, but in different ways, and with different domain-specific advice for the two specialized versions.

Unfortunately, all the Swiss Cheese holes had all lined up this time, and twenty porcupine quills had come through those holes and into Brownie’s muzzle and nose. :-(

The Shoveller: We’ve got a blind date with destiny… and it looks like she’s ordered the lobster.

But what to do now, a twenty-minute walk from home, and from there another fifteen-minute drive to our (almost certainly closed already) vet, and a forty-five minute further drive to an emergency vet. I knew also, in any case, that my wife and son were out in our one car, likely for another two hours. No other houses were between me and home to ask for help. I guess in retrospect I could have gone back the other way about ten minutes or so to the house with the Jason mask. I might have considered that if I had known them well or found the place less off-putting given the year-round scary Halloween decorations. I thought of another neighbor who was a nurse and had dogs, but her house was still about a thirty minute walk from where we were. Meanwhile, Brownie was pawing at her muzzle, flicking her tongue round with an almost hissing sound, and starting so get agitated in other ways. I tried to start walking her home for a few yards, but she kept pawing at her face and it seemed clear that walking her for any length of time in that condition was not going to work well.

So, I held my dog close and started looking at the quills. Thankfully she did not have one in her eye. Guessing the quills probably had harpoon like barbs, I wondered if maybe I could at least push some of them out through loose skin. I tried that with one furthest up her muzzle, but that did not seem to work, and the quill even broke off in the middle. So, I tried pulling the quill out real fast holding onto the remainder, which seemed to work. I did not know if anything was left stuck under her skin though from the head of the barbed quill, but it seemed the best I could do. I knew it would be best to get someone more knowledgeable and with the right tools to help, but that did not seem possible anytime soon. It just seemed to me that leaving Brownie with all the quills in for at least twenty minutes (to get home) or more (to get somewhere else) was not a good idea, as she continued to paw them and flick her tongue at them, and she even started to shake almost like having a seizure.

So, holding her close with one arm and hand, I just started pulling them out one by one with my fingers of my free hand. I later read on the web how pulling quills with your fingers was not advised as they can have side barbs too, and my fingers did feel a bit odd a while afterwards like having handled nettles or such. I did have a small pair of tweezers in my pocket knife, which I did not think of using at the time, although I can wonder if trying to use those small tweezers would have made things harder. Likewise, I could have used a dog waste bag as an improvised glove, but I did not think of that at the time, although that also might have made the task harder.

Removing the fifteen or so quills in Brownie’s muzzle went OK, with me learning more about pulling them as I went — but the five or six quills in her nose I left for last. I guessed they would be the hardest to pull, and I still had thoughts about getting Brownie home before trying to pull them. But as she still pawed at them and licked at them, I did not think I could just leave them and try to walk her home. So, I pulled them fast too, one by one. Each one was a struggle with her, all while the light was fading with sunset. The last few took longer than pulling all the previous ones as Brownie bobbed her head around. At some point I had my hand half in her mouth to keep her head steady. The last cluster of three was the hardest as by then, after pulling a couple of quills out of her nose, she was very hesitant to let me do more. All the quills must have hurt as I pulled them, and her nose was obviously the most sensitive spot. Finally I got the last one out and she seemed relieved. After we got home, I put some colloidal silver over the area (as well as on my fingers, where I also used some oil of oregano too). I fed us both some extra dulse (high in iodine, which may help repel infections) and did some other things. I looked Brownie over carefully with a flashlight for more quills, including quills in her mouth, as web sites on the topic advised. As I write this the next morning, I may still end up taking her to the vet, just in case, although she seems happy enough right now curled up beside me as I write this on a laptop sitting in an oversized chair.

It was in a way a miracle she did not bite me during the quill-pulling process. All dogs can bite under enough pain, the only question is, how hard will they bite? When we first got Brownie as a puppy about two or three months old, she had ended up biting my son on his ankle at one point while running alongside him (enough to draw blood and give him a small scar). She never bit out of aggression; she just seemed to think it was part of playing or greeting that you bit someone, probably a bad habit she picked up somewhere as an unwanted puppy shipped up north from Georgia. We could have brought her back to the animal shelter perhaps (to who knows what fate as a “biter”). Instead, I spent the next month or so doing Ian Dunbar’s bite inhibition training with her to at least give her a soft bite (training that really only works with puppies, not older dogs). That training entailed putting her on a lead connected to a post in our living room, playing with her to get her excited, and then backing away to end the play when she bit hard on my hand, gradually lowering the amount of force I would accept before ending the play. Only after you shape a “soft bite” do you then try to get the dog to bite less often. I don’t know what she might have done without that training as a puppy, but I can only guess it was helping us both these two years later. I am so thankful to Ian Dunbar for his writings, as well as the blogger who wrote about Dunbar and thus helped me when I turned to the web in desperation to figure out ways to deal with our puppy’s unwanted behavior. (I use the walking-away part of that technique with our adult cat as well when playing with him, and he definitely scratches me less while playing than in the past.)

During the rest of our walk home, Brownie seemed to be acting like nothing unusual had happened, splashing through a stream in a ditch by the side of the road, and jumping around and smelling things as she usually did. I planned to look up advice on the web on porcupine quills as soon as I got home. But I also thought about how if I had called my neighbor (the nurse with dogs) for advice or help it would have in a way been a community-building thing even as I did not want to bother her. (My wife has helped her in the past with a small non-profit related to animals she wanted to set up, so aid goes both ways.) The web may be a great resource, but there are still many aspects of the web related to community that could ideally be improved. Or, maybe perhaps the web never can fully have the quite same feeling of community, even with “likes” and “thank you” comments and so on — in which case at least the web can still help organize or otherwise support face-to-face communities.

After all, and sad to say this, but I never thought at the time to thank the blogger who may have saved Brownie’s life (compared to returning her to the shelter) and who had helped keep me, my family, our friends, our neighbors, and random passersby from being bitten. At this point, not even remembering the blog found when searching, I can mainly just “pay it forward” into the gift economy and help others. Although, as with the Happiness engineer “thank you” feedback suggestion above, that makes me wonder if maybe WordPress blog posts could just have a simple “thank you” button on them in addition to comments? Saying “thank you” is not quite the same as choosing “like” or “dislike”. And a “thank you” also is not even quite the same as a “comment”.

Rabies is apparently a very small risk from porcupine quills, although the porcupine seemed healthy enough, thankfully, from what little I saw of it and knew of its likely habits. So, I’m not necessarily recommending what I did — I’m just explaining my experience as a story. My dog has already had a couple of rabies vaccinations, but there is a very small chance I will die from rabies from what I did. Yes there are precautionary rabies treatments, but with a big health insurance deductible for us (the cheapest plan), it’s a tough call if such expensive precautions are worth it for me just from having pulled about twenty quills by hand (as opposed to, say, a greater risk from having received a direct bite from a strangely acting bat or whatever). On the plus side, after you get such treatments one blogger wrote he became a kind of superhero helping neighbors deal with potentially rabid animals for a while. :-) If I had been closer to home, with a car available, with another person like my wife around to help, during the day with an open nearby vet, I would likely have done things differently, especially with what I know now. I’ve also since added a disposable glove to my dog training bag. My family and I may yet pay a steep price for that accident caused in part from my inattention and lack of preparation (as well as other factors like the design of the plastic buckle clasp, and also the attentiveness of some plastic-handling factory workers in China I have never met and who I did not until now realize my life may have depended on). Of course, every time we get into the car, it’s a risk too of a car accident. Risks can’t be totally avoided, just minimized and weighed against costs and benefits. Not getting outdoor exercise is a risk too. As is not thinking creatively about various challenges in our lives.

By the way, I do not harbor any grudges against the porcupine for defending itself from an inquisitive dog. We were the ones trespassing around its home. There are deeper things one can learn from porcupines as well, like from this insightful blog comment I found while using the “DuckDuckGo Health Plan”: :-)

Porcupine sometimes operate as a natural anti-rabies mechanism. I’ve seen more than one colonial report that mentions dead rabid foxes and wolves with faces full of porcupine quills. Mindless aggression and biting has certain drawbacks…

By coincidence, as the relevant point of this whole long story, what my mind had been wandering towards while a porcupine’s tail swiped Brownie’s muzzle was disaster recovery (DR) planning at Automattic. :-) That destination was primed no doubt by my walking earlier past that house where the tragedy happened that so many people would so much wish to have prevented.

I had been tangentially involved with one DR setup at NBCUniversal, and of course over the years had taken care of my own (literally) in-house systems. I had been thinking about a question I might someday ask Matt Mullenweg (or others at Automattic): “What would you do if at this very moment every WordPress website in the world went offline?” Just step by step, think it though… What would you do?

Naturally, that’s not the kind of thing I’d really ask my first day on the job. People might think I was prying into too much too soon, given I would not be hired as a DR specialist. Also, neither am I experienced enough in that specific DR role to do more than kibitz about it. It’s just the kind of thing my mind was wandering towards while approaching the old barn and our ill-fated not-quite-Halloween meeting with the porcupine.

The simultaneous complete failure of all WordPress sites in the world is of course far fetched. But, given Microsoft’s experience rolling out a bad update that started bluescreening Windows boxes, it is still not beyond the bounds of possibility. Just one bad line of code that got into a WordPress automatic update could potentially have that affect either at the time of the update or days later (a good reason to automatically test WordPress updates also on virtual PCs with clock dates in the future).

While it would be inappropriate to reveal the specifics of what little I know of NBCU’s DR plans, the general issues of having multiple sites and multiple ways of doing things and of keeping multiple humans in the loop in multiple ways are the basics that any company needs to consider. It’s essentially the same sort of idea that is mentioned above in that document on system fault tolerance and in the Wikipedia page on the “Swiss Cheese” model. Again, while not a DR specialist, based on those sorts of generalities, I’d be more than happy to kick around ideas about what to do if the porcupine ever strikes at Automattic. :-(

From over a decade of reading DR and computer security discussions on Slashdot (often based apparently on first-hand experience by the posters), it seems that many computer security and DR people say you can’t prevent all intrusions and you can’t prevent all equipment failures — no matter what you do. The porcupine is going to strike sooner or later. You should of course take steps to minimize the frequency of such bad events (like by being more watchful walking near old barns that have porcupine quills on the ground around them). However, what you most need to do is have a plan in place to get the quills out and get back in action as quickly as possible when (not if) the porcupine strikes.

The web is rapidly becoming the repository of all written human knowledge. It is amazing that Automattic supports a quarter of the web with so few people. It is an awesome responsibility to ensure all that knowledge remains accessible.

WordPress has a wonderful export/import feature, but it is not always used, especially as it is a bit out of the way in the admin menu and thus non-obvious. Beyond its own internal DR plans, Automattic might want to add a feature to WordPress that nags users to do regular backups via export every once in a while and tracks when then latest backup was done. An export still does not include media, but there are other options for that. Or maybe export could be further enhanced, like to stream everything to another very different archival system such as a CouchDB-base one, maybe as a partnership with Cloudant or such to get vendor redundancy? Of course there is also VaultPress as a great product, and so the nag screen might also make it easy to sign up for VaultPress. :-) However, even with VaultPress, there still is almost nothing as reassuring as having decentralized backups (both remote and local). There is always the risk some porcupine disaster may strike in the “cloud” or at a specific company despite the very best plans and testing of everyone, where the refrain will be: “It all happened so fast!” :-(

Disasters happen, often when you least expect them. That’s one reason people buy insurance or self-insure via savings if they can possibly afford it. That is one reason people train for such situations too (including perhaps in broad ways like teams having non-coding social meetups to learn trauma care or executive-level defensive/evasive driving mainly just to sharpen the team’s reflexes and sense of togetherness — even if you never know when such skills live first aid or evasive driving might actually prove useful). One of the people I worked with at NBCU had previously worked at a bank-related computing complex in another country that apparently was invaded by people with weapons (thankfully for him not when he was there, but he said a colleague was almost killed).

After such a disaster is when people will ask tough questions about whether Automattic did everything reasonable to safeguard the accessibility of a quarter of the web (and rising). While DR is not the sort of thing I’d like to focus on at Automattic, I’d be interested in helping out with discussing DR now and then to provide one more perspective on the topic.

Of course, let’s hope any plans for coordinated human action for a situation like all WordPress sites going down simultaneously never need to be put into action, or if they do, it turns out to be a false alarm. :-)

[Mr. Furious, the Blue Raja, and the Shoveller are sneaking across Frankenstein’s property to undertake some reconnaissance]
Mr. Furious: Shh! Wait! Hear that? We must have hit a trip-wire. It could be a proton eviscerator heating up.
The Shoveller: It could be a cybernetic atom scrambler. They target moving objects.
The Blue Raja: It sounds more like a heat-seeking anthrax projection gun to me. Quickly, cover your mouths.
Mr. Furious: No, let’s bolt! Come on!
The Shoveller: No! Don’t move.
The Blue Raja: Don’t breathe!
Mr. Furious: Let’s bolt!
The Shoveller: Don’t move!
The Blue Raja: Don’t breathe!
Mr. Furious: Let’s bolt!
The Shoveller: Don’t move!
[a sprinkler system turns on and the three are soaked]
The Shoveller: [disgustedly] Ohh, it’s the sprinkler.

Of course, an alternative interpretation of all this is to be careful what you wish for, as in, maybe I’ll end up getting a nose-full of porcupine quills from writing this long essay in a rush try to get a job at Automattic? :-)

Automattic and in-group/out-group exclusivity?

Here is another thing I’ve mused about while out walking the dog on a walk that thankfully did not end in quills that time.

Automattic’s audition/trial process obviously is working well. It was a great innovation to bring an idea from the formation of musical teams into the formation of programmer teams. It’s not the first time programming has benefited from musical culture, and likely it will not be the last.

However, here is another thought experiment of taking things to their logical conclusions and beyond.

I’ve heard that Bill Joy (co-founder of Sun Microsystems, as well as author of the essay “Why The Future Doesn’t Need Us“) would often ask the same question of any idea proposed to him: “And then what?”

By the way, it’s a fair question to ask of the Twirlip proposal idea above. And I’m not sure I have a great answer, other then a hand-waving one that the world eventually hopefully becomes a more joyful place through well-reasoned arguments and better clarity about the emotional motivations behind the arguments.

So, just for fun, let’s take Automattic’s success and policies to the (illogical) extreme and say, what if Automattic not only was hosting 100% of the web, but everyone worked for Automattic as well?

Let’s assume everyone gets issued an Automattic ID card at birth (or even before), and gets a good salary as a basic income from day one, and they could not be fired. Let’s assume all manual labor was done by tele-operated dumb robots controlled via JavaScript-powered interfaces hosted as various WordPress plugins.

So, the whole world is Automattic and WordPress. :-) And then what? :-)

I don’t have an answer to that either. It is just interesting to muse about in relation to Automattic’s growth and related challenges.

In such a scenario, you can’t rely on auditions to limit who is in Automattic, because everyone is an Automattic employee. Nor can you fire poor performers. Matt Mullenweg’s email load would likely have finally moved beyond that of Santa Claus, :-) so his time would be very limited to help with the new demands.

What would Automattic do in that case with seven billion new employees? I don’t know. It’s just a question to reflect on. A few of my own reflections are below.

When I was an undergraduate at Princeton, I took music lessons for flute, which I had recently picked up. But the lessons were expensive (even if more than worth it). You heard you could get a 50% discount if you joined the Princeton University (PU) orchestra (the Band did not count). So, even though I did not especially want to be in the orchestra, not having much spare money, I auditioned. It did not go well, and I did not get in (not that I was disappointed much, I was busy enough as it was). It was a fair assessment of my abilities on the flute, even if it made it more likely my flute lessons might have to be curtailed.

By contrast, when I joined the Princeton University Band (a fun-loving “scramble” band in contrast to precision marching bands), there was no audition. I became essentially a “trash” flute, like the “trash” percussion section — “trash” was the word we all laughingly used and no one felt bad about it, as trash players were just happy to be included. I just mainly added to the number of orange-and-black-clad people on the field running around and doing funny formations — often sarcastic or edgy ones that could just barely be slipped past the university censor. Beyond having a flute, I also had a piccolo I had gotten as a gift from my parents, and I could just barely help out a bit with the Stars and Stripes (otherwise carried by our flute section lead who was the only other player with a piccolo). I had a lot of fun in the Band, got to visit other college campuses and made some good friends (even if the drinking seemed excessive). Compared to the PU orchestra, the PU Band was a model of inclusivity. There was room for everyone.

However, in thinking about that experience in an Automattic context, I now see that I have overgeneralized the Band’s inclusivity. While it was indeed inclusive, it was only inclusive of Princeton students. These were people who had already been highly filtered already on things like conformism, assignable curiosity, unquestioning immediate compliance in the face of unearned unaccountable authority, and all the other things high grades in most schools select for according to John Taylor Gatto.

As an admission of deep previous naiveté, my entire time at Princeton as an undergraduate, I always thought of “exclusive” as meaning “excellent”. It had never occurred to me once during that time that “exclusive” meant “to exclude”. Sad, but true.

Internally, based on Scott Berkun’s book, Automattic seems highly inclusive, like the Princeton Band in that sense. Company blogs are accessible to all, anyone can get involved and help out with some distant project, and so on.

But that Automattic inclusivity also depends on a PU-orchestra-like excluding of “trash” players at the start. No doubt many good players never get to audition for the Automattic orchestra for whatever reasons — overly long essays about Automattic’s potential future being number one on that list of reasons no doubt. :-) Or, they might flunk their auditions for whatever reasons, even if maybe someday with more time or training they could pass it.

Like Princeton University’s admission committee, Automattic tries to hire people who already match the culture. That is as opposed to bringing in random people and trying to help them learn the culture and find a place for them in it, however great or small their abilities (like, say, during a military draft), while also risking they may change the culture (including potentially in negative ways). Obviously, Scott Berkun was hired specifically to bring new ideas to the Automattic culture, so this is not intended to imply Automattic does not change or does not want to change if needed like from growth issues. It’s just an observation about what seems to be, given an audition process based on a certain set of communication tools and self-management assumptions — and I’m not suggesting that should change.

I have no doubt an acculturation process still goes on with new hires at Automattic. I read somewhere it can take a year to really find your feet. But whatever goes on with new hires after a successful audition, it is nothing like what would go on if Automattic employment was open to people of all ages and all interests and all abilities (let alone required of all people as a service obligation).

By contrast, a place like SUNY Stony Brook was a lot less exclusive than Princeton, taking anyone with a B average or higher in high school, plus also making lots of other exceptions. As John Marburger, the president of SUNY Stony Brook at the time, and also a Princeton alum, said to me while I was there, it is easy to measure a college by what it turns out, but what you should really measure a college by is the difference between what it takes in and what it turns out. He was a physicist, so he was no doubt comfortable with differential equations about differences. :-) Princeton takes in the best students and turns out people not that much different in that sense, even if they may have always scored well in certain ways. He felt SUNY Stony Brook made a bigger difference in improving people’s abilities and lives. Having attended both places, I tend to feel he was right. Stony Brook was also just a more diverse place in many ways which had many benefits, even if Stony Brook also had its share of problems from that. There may be much better alternatives than college for most people, but as far as college measurement goes, his was a good insight.

Again, I am not implying by any of this that Automattic should change its approach. Things do not have to scale. For example, NarraFirma’s current approach of downloading all the stories for local processing each time the webapp is started can’t scale past 1000 stories or so. But since most Participative Narrative Inquiry projects are only a few hundred stories, that is good enough for the intended purpose. All living cells have filters surrounding them as semipermeable cell wall membranes. Automattic may well be the best it can be at its scale — even if it may also end up suffering from some of the same issues Princeton drifts into through a certain lack of cognitive diversity through whatever filtering process it uses (even just filtering for the “best and brightest”).

WordPress itself is very inclusive though. Anyone can download WordPress and set it up at no cost for the open source code from WordPress.org, or they can get a friend to help with that, or they can just host at Automattic via WordPress.com.

Even if Automattic is a lot like the exclusive PU orchestra, WordPress is a lot like the inclusive PU Band — but without needing the first exclusive layer of college admissions filtering.

One limit to auditions is that they assess people at one point in time, and people change over time for all sorts of reasons. People may learn new things and forget old things. Their family situations may change (for the better or worse as far as distractions from work). Their life partners may change jobs or roles causing follow-on changes in their own lives (as in my own case). Their health may change (both ups and downs, like as they get more or less vitamin D from outdoor sunlight or supplements, or eat more or less vegetables). Their interests may change. Their willingness or ability to travel or to stare at a screen much of the day may change. Or, even more confoundingly, the organization’s needs may change over time, as when IBM Research hired the best mechanical engineers in the world because of the success of the IBM Selectric typewriter, but then later found many of those engineers became unhappy when asked to become programmers. So, in that sense, auditions do not solve all problems of management, even if they make managing a technical organization easier at the start and likely improve overall results.

Although, of course, whatever you filter on will likely limit cognitive diversity in some way. It’s been said, for example, of Google’s earlier hiring practices that they selected for people who liked to solve complex problems, and such people might not be the best for intuitively assessing the benefits of easily approachable user interfaces. For Automattic, people who are very comfortable working through the web may just, sometimes, have a harder time getting into the mindset of people who are not comfortable doing that.

I’ve learned a lot from hanging around with other people from my local historical society, most of whom are quite a bit older than me. One photographer, more than ninety years old (then our secretary), showed me a thing or two about digital photo editing on her computer. In general, I’ve enjoyed our board meeting as something non-computer-focused to do, so I’ve actually been more of a foot dragger as far as digital automation compared to what some other board members might wanted to do. I certainly didn’t win many points by not being a Facebook fan. It has also been sad to think about the board’s face-to-face meetings likely eventually passing away, or becoming less pleasant if people Skype in with lag (as was proposed for a possible new board member). Even the off-the-cuff locally-educational side discussions have perhaps changed a bit since the last two secretaries started recording the minutes digitally (something that happened with no substantial prior discussion). Jokes that are respectfully made in a context of good will may sound different by themselves. We on the historical society board all agree what a pain in the backside computers can be — although my own feelings on that topic may be a bit more complex and for different reasons. :-)

Some of my reluctance to “computerize” the society is also from limited time though, even beyond other concerns about who would maintain or keep secure whatever I do. That is why I chose WordPress.com ultimately for our upgraded website instead of, say Drupal, or something hand-crafted by me with Node.js running on a VPS somewhere. If I had more time, I still think it would be cool to construct current and historical models of the town in Minecraft (or better, Voxel.js), for example, as a way to engage a younger generation in local history. Maybe someday. So many stories are “local”. Connecting stories to a walkable landscape is in that sense is a very human thing to do. Our narrative memories may even be adapted to work that way, with the landscape reminding us of things, like I was reminded of stories and themes when passing that tragic house mentioned earlier. Now I will likely be reminded of porcupines, Automattic, and disaster recovery when walking past the old barn. Linking WordPress stories and a 3D landscapes might be a fruitful thing to explore, whether the stories are inspiring or cautionary. But thinking up ideas like those only comes, as one needed aspect of many, out of a variety of diverse experiences.

There are ways to deal with a limitation of cognitive diversity in any organization, of course. It sounds like Scott Berkun was making a great start on that at the end of his time at Automattic with his work on promoting a more formal GUI evaluation process and perhaps more systematic experiments. And of course, Automattic has other ways of assessing GUI usability from posting and commenting numbers and such. Something like NarraFirma and Participatory Narrative Inquiry might help with that as well, since 200 or so stories collected from WordPress users about their recent experiences with WordPress might provide some unexpected insights into actual user needs. Of course, it’s always a gamble on whether you’ll see something new with a user survey depending on how questions are framed, what you already know, response rates, who actually chooses to respond, and so on. Everything has its limitations. As with the “Swiss cheese model of accident prevention” mentioned earlier, you can hope that with enough diverse human layers with perception gaps in different places that no big issue slips through your company.

This section is more to frame a question about Automattic’s growth and management over time than to provide an answer. However, as a suggestion, it might be worth exploring more ideas from musical culture to see in more detail how musical bands form, change, and disperse, and to learn more about how people come to join bands and leave them. That’s not the only way to explore this issue, and I know next to nothing about that, but it is a place to start, and it is in keeping with the musical audition metaphor.

Ideas about stigmergic and chaordic processes might help as well. However, I don’t know specifically right now how to translate those general ideas into specific plugins or specific social processes for an Automattic context. That’s an issue that would take a lot more thought, discussion, and experiment.

The Spleen: Hiya fellas. Word on the street is you’re lookin’ for superheroes.
Mr. Furious: Not really.
The Shoveller: Actually, we’re just leaving.
Blue Raja: Yes. Rotten luck.
[The current Mystery Men stand up to leave, but then sit back down then the waitress suddenly arrives with their food.]

Can you help with any other needs at Automattic?

One other thing Scott Berkun says in his book is:

“Automattic hired many successful freelancers but few were product thinkers — people with visions for entire product concepts. Raanan Bar-Cohen, the lead of Team VIP Services … often pointed this out as a cultural gap. He felt the company needed more people with entrepreneurial instincts inside the company.”

Assuming that gap still exists, given my experience working on several products from start-to-finish, and my passion to make more software products in various areas including public intelligence and education, I can help fill that gap. I would enjoy helping the Automattic community create some more “really useful (JavaScript-)engine” products through taking part in the stigmergic and chaordic processes going on at Automattic. :-)

Still, I would not call myself a successful entrepreneur (other than in the sense of having made some free software products while keeping a good credit record of on-time payments). But I also did not sadly die of a heart attack at an early age like this guy (after becoming the youngest alum to endow a professorship chair) who lived across the hall from me my first year at Princeton. And I did not turn people into robots during the day like another guy from that Princeton class (even as he may be planning to replace these people with real robots) — even though we still buy much stuff through Amazon with mixed feelings. Of course, I could have been a lot wiser in many of my own choices even while avoiding those extremes. Live and learn. :-)

Could you sum up your aspirations about improving WordPress and why Automattic is a good place to help make that happen?

As a long-time Smalltalk fan, I still hold on to aspirations to perhaps someday build something better than today’s PHP-based WordPress to support comprehensive information management and sensemaking software.

Here is a grandiose video I put together about Twirlip when I was considering making a Kickstarter about it back in 2011, but I decided not to proceed with fundraising for it that way.

Here are a few key facts I have considered in the years since making that video:

  • The possibility exists to build a better WordPress in JavaScript on top of PHP+WordPress+AJAX/REST/JSON.
  • An opportunity is there to help the huge existing free-software-emphasizing developer community for WordPress move in that direction to create endless innovative JavaScript-powered plugins for WordPress using newly-emerging core AJAX/REST/JSON APIs — which most likely would among other things include a set of public intelligence tools beyond what I or anyone else can imagine (even being imaginative people).
  • Automattic has already had success in creating an innovative community that can offer paying jobs to develop open source JavaScript software (as well as facilitate the development by others of such software) to take WordPress in new directions to make the web a better place in all sorts of ways.
  • Automattic is a company I would have been proud to have created as far as its policies, products, and sense of community.
  • With Automattic’s continuing growth, there is likely a need for more data Janitors to write tools that help everyone at Automattic, and at least some of those tools could be JavaScript-powered WordPress plugins that anyone might want to use, and many of those ideas could be steps towards the ideas I outlined in the Twirlip video above.
  • At this point in my life, after assessing my own strengths and weaknesses and track record in different situations, and given family and local community obligations, I’d much rather join and support a good community like Automattic/WordPress interested in incremental evolution than to try to build such a community from scratch — even if I realistically could (which I realistically can’t). My father worked many different jobs throughout his life, at first going from ship to ship as a merchant mariner and Bosun. Then, after getting married, he did machining and shop supervision at different small companies as they got big contracts and then had layoffs when the contracts were done. However, the last ten years of his career were spent at just one mid-sized company doing interesting work with a great crowd of people he liked. Working with a good group of people for a long time seems to me a sensible choice towards the end of an otherwise diverse career. By then, you know from experience that the grass isn’t always really greener somewhere else, however it looks from afar. And you also learn, as my world-travelling father used to say, “Wherever you go, you take yourself along.”
  • Practically speaking, now that my wife and I have blown all our cash on making yet more open source software, :-) and even as we hope that investment in NarraFirma may pay off in the long term in enhancing my wife’s career and helping the world, in the short term, given our present economic system, I need to find a job. And given that we chose to buy a house in a rural area, working mostly remotely is a big plus for me, as less time spent commuting means more time for friends, family, community, exercise, hobbies, and personal growth.

So, given all that, it seems, “If you can’t beat ’em, join ’em.” :-)

Of course, ironically, with likely long email backlogs at Automattic (including both for recent significant growth and a Grand Meetup), by the time anyone at Automattic reads this (or parts of this), I may well have taken a job elsewhere. :-( And then there is my wife’s point that this essay about me and Automattic’s future (now almost half the size of Scott Berkun’s book about him and Automattic’s past) is way too long and rambling. So, she may be right that this document can be easily taken as proving me unfit to work at Automattic. Writing at length is at odds to a web of communications increasingly built from 140 character messages, such as Scott Berkun describes even at Automattic. This document was based on my reading a lot about WordPress and Automattic, so it is an interactive in a large-grained sense. :-) That is in contrast to finer-grain interactions like happen on IRC, Twitter, or Skype chat. (I can send those too, by the way; see my Twitter feed. :-) And, as with Brownie and the porcupine, a hound-like chase of certain ideas and a terrier-like persistence in approaching them may just get me a nose-full of quills in the end. :-( Maybe I would not even really like working with Automattic? Maybe things like public intelligence tools will just be too hard for me or anyone else to make even if otherwise everything else is great — and so some other choice of project would have done more for the world (including just more work on a more focused NarraFirma)? So, things may not work out for various reasons. Even if so, I’ll still consider the couple of weeks or so I spent writing this (sadly to my wife’s distress, instead of looking for other jobs, as I must finally get to right now), as time very well-spent if it can help a great company like Automattic make the web an even better place to be — even without my direct help. After my wife and I released our garden simulator and related educational materials onto the web for free around 1997, we used to console ourselves as we dug out of debt working at IBM Research that we had gotten the web in return. :-) This (by now) short book about a potential future for Automattic and me and the web was a “gamble” — or, more positively, an “experiment”. :-) Maybe it will someday pay off in benefit to me, Automattic, and/or the web (as well as the larger world around all that). Maybe it won’t. In any case, all the best.

–Paul D. Fernhout
====
The biggest challenge of the 21st century is the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.