Inner Source : know it’s limits

For the last few years, along with colleagues at Red Hat, I’ve been of the belief that Inner Source (see Wikipedia definition) is a viable alternative when an enterprise organisation doesn’t want to embrace a full Open Source approach (for a variety of reasons).

The differences between Open Source and Inner Source are listed below, but the main difference is that former usually involves more that one organisation and the latter is very much internal to one.

Open SourceInner Source
Not always a formal organisationMore likely to see some form of top down formal organisation
Open flow of information and ideasSome control and ownership around information flow
Meritocracy and Benevolent DictatorshipSome management direction of a project
Social phenomenonInduced social phenomenon
No budget and time constraintsBudget and time allocations needed
Potentially infinite number of developersFinite resource and input
Consumable by allLimited audience for consumption and participation
Voluntary teamsPartial voluntary, with allocations
Self assignment of tasksAllocation of tasks, resource
Low consideration for competing tasksConstant need to consider alternative tasks and activities

Inner source is like Irish dancing, you are in a constrained realm. Open Source lets you fling your arms about like a Scotsman.

Malcolm Herbert March 2019

There’s no doubt that having the barrier between an organisation, however large it is, and the outside world is a significant barrier to the success of inner sourcing projects. Inner source projects need constant nourishment as they are not getting nutrients from the outside world.

Irish Dance

Red Hat’s Office of Technology in Europe does a range of consulting work around open source process and methods, helping organisations leverage the model successfully used to produce software over the last 25 years.

In the past an Inner Sourcing approach has been adopted for organisations, where either:

  • there is need to keep the activity separate from the outside world
  • that the project isn’t primarily software development

However, the effort of maintaining and sustaining an Inner Source environment isn’t repaid. Remove funding or support and it dies. All very Darwinian and while these extinction events also happen with Open Source projects they aren’t so dependent on internal or commercial factors.

Inner Sourcing approaches always run the greater risk of duplicating the efforts in another organisation (and the same mistakes) if the work is made widely available. So what’s the downside to Open Source and letting your team collaborate outside the corporate structure ? Being Open Source potentially needs a governance structure that cover what people can and can’t do and understanding of how it works in order to draw these guidelines up (it might be why you were looking at Inner Source initially). There are lots of sample Governance structures out there and some good articles.

Persuading an organisation to look beyond it’s own boundaries is getting easier as the recognition of the success of open source methods and process is becoming more widely understood. Whilst the consumption of open source (both community and enterprise) is the norm, collaborating and creating with upstream community projects is becoming an option for more organisations, as developers and managers have positive experiences of working as part of open source projects. For an increasing percentage of developers it’s the only way they work.

Based on the work Red Hat has done with organisations, the benefits (obvious and otherwise) of going open source include:

  • the usual coding quality benefits (many eyes etc); you have a bigger community externally
  • freedom for developers to interact and be creating with other like-minded developers
  • developer visibility (they like their GitHub commit stats and associated kudos)
  • developers hiring and retention is easier with Open Source activity and practice in place; Inner Source is less appealing. This is becoming a key benefit now that Developers are King (see O’Grady et al)
  • that many organisations don’t do all (or any) of there software development in house; using an Open Source approach does lend itself to managing multiple software suppliers (standards, ways of working).

And this last point is interesting. Some organisations, that don’t do a significant amount of software development, feel that Inner Source is the only way for them to experience the open source benefits. However, if they started running their software providers as a community then issues over lock-in, poor product etc are less likely.


A40 Brewing : Brew #2

After the failure of the mild, the second recipe was a low ABV India Pale Ale, primarily using Fuggles hopes for brewing and potentially then dry-hopping using Goldings (or maybe more Fuggles). Not exactly normally, but I’m not a big fan of over-hopped IPAs. More like a bitter then….

08h00 prepping the starter. Wyeast Whitbread Ale smack pack with 1 litre of DME mixture
15h00 : Finished crushing the malt. 5kg of Maris Otter pale malt and 700g of Caramalt in the grist. Have to the adjustment for the crusher not too bad but whilst precrushed malt is easier, is it fresh. Again this batch came from The Malt Miller
15h30 Started the boil about 37 litres of water in the HLT and aimed for a strike temperature of around 78 C based on the grist weight and temperature.
16h20 : Started the mash with an initial temperature of 69 C after adding about 17 litres to the mash tun. No real clumping of the grist but stirred it to drop the temp to 68 C and ended up at 67 C ! By 17h20 the temp in the tun had dropped to 66C but had held the temperature pretty well.
17h20 Mash just prior to sparging
17h30 Verlauf, took about 7 litres from the mash tun and then poured gently back into the top of the grain bed to get it to run a bit clearer before starting the sparge.
17h45 Sparging, tried to do this slower than we’d done it for Brew #1 with the aim of a 1 hour sparge. Ran the tap really slowly then added water from the HLT (at around 75 C) into the top of the mash tun using a 1 litre pyrex jar and a collander. Automating this process with a sparge arm would by good, but would need to look at gravity set up which would mean raising the HLT well above the current bench.
18h00 : Used the refractometer to measure the gravity during the sparge. Based on the recipe calculations was looking for an OG of 1.037, with around 30 litres of wort to boil. After 10 litres was at 1.065, 15l 1.050 and then at 20 litres 1.043. Measurement at 25 l was down to 1.036, so very quick drop off and 5 l short of the expected volume from grist weight.
18h10 : Boil started. The Sparge only took 40 minutes, again too quick and maybe the reason for the lower than expected OG / wort volume.
18h27 : Wort starts to boil and added 30g of Fuggles hops. Added another 30g at the end of the 90 minute boil and let rest for 30 minutes before cooling.
20h30 : Cooling the wort through the heat exchanger. Need to run the wort through twice to get to 20 C in the bucket. Did look at running straight into the fermenter using the pump, but this produced a high pressure leak on the pipe straight away so went for a much simpler approach of half-filling the bucket and pouring it into the fermenter.
21h00 Wort into the fermenter and then oxygenation for 3 minutes. At 5 PSI and using a 5 micron stone dropped in to the wort. Then pitched the yeast starter and sealed the fermenter.
08h00 Next Day. Much better looking krausen than Brew #1 and quite a bit of noise from the airlock. Need to ensure that the fermenter lid is on tight as it will leak otherwise.

A40 Brewing : State of the Mild

The mild got binned after been stuck at 1020 gravity and with no real fermentation taking place. Dried yeast was added after 5 days in attempt to get it going but no luck in getting it moving. There was no real head / krausen formed on top of the wort and no movement through the airlock.

This was potentially down to a number of things:

  • lack of air / oxygen in the wort. Whilst we did pitch it from a bucket into the fermenter, it made have needed more swirling around after the boil
  • the mash didn’t go well and whilst the temperature seemed okay at around 67-68 C, potentially not enough sugars etc were extracted in to the wort.
  • the original yeast starter didn’t really start. Wyeast 1099 Whitbread Ale liquid yeast but it didn’t look right when it was made an pitched.

It was Brew #1 and though not successful, plenty to think about going into Brew #2 and some changes in the process. Low 3.6% ABV IPA coming up next.


A40 Brewing : Brew Day #1

Crushing the malt; make sure the battery is charged on the drill. Cheap Chinese made crusher but is fully adjustable and seems to work well.
  • 16h00 : Start Wyeast Whitbread Ale #1099 in 900Ml Water + Dried dark malt extract at 80:20 water:malt. At 20C and in flask over agitator. Maintained agitation until time to pitch (about 4hr45)
  • 16h00 : Start heating 33L water to 80C (We did 78C but 80 better)
  • 16h50 : Begin Mashing: 15L water. 4.5 Kg Grist as above. 78C water gave 68C Mash
  • 17h10 : Temp drop 2° in 20min so added 1L at 75C to 16L total now back at 68C
  • 17h15 : Start reheating plain water to 80C for sparging
  • 17h40 : Mash temp down to 66C
  • 17h50 : Begin Verlauf. 1L wort at at time. Down side of mash till wort started to clear
  • 18h00 : Finish Verlauf
  • 18h05 : Begin Sparge. 2L water at a time at 80C. Add about 6L. Tot vol only 20L (due to soaking into mash and evaporation).
  • 18h30 : Test sample wort at approx 20C. OG 1045. Sparge further to 29L total. Wort now runs clear
  • 18h40 : Re-Test. OG now 1035. Total Sparge water was about 15L.
  • 18h45 : Move copper to heat and turn on
  • 19h10 : Rolling boil @ 100C. Add 50g Fuggles Hops
  • 19h40 : Add Irish Moss (Copper Finings)
  • 20h40 : Crash wort into sterilised bucket for transfer to fermenter. Heat exchanger with steady cold flow and slow flow of wort drops teml from boil down to about 16C.
  • 20h45 : Transfer cooled wort to fermenter and pitch yeast. Remaining Vol 26L. Fermentation fridge set to 20C. Ambient temp in brewery started about 10C and rose to 15C.
Grist; 4kg mild ale malt, 420g Crystal Malt, 120g Chocolate Malt from The Malt Miller in Swindon. Brew Day #1 was to make a mild, as per the recipe which is uploaded into Brewers Friend.
The HLT above the Mash Tun. Gravity used for the brewing rather than the food grade pump shown (will perfect the setup in subsequent brews) . The 50 litre brew kit came Powell Brewing in Flint, North Wales. Build quality is excellent and even with my recent lack of brewing experience, things seemed go well with the kit.
Yeast starter. Made a 1 litre starter using Dried Malt Extract as the base and with liquid Wyeast 1099 Whitbread Ale yeast.
The mash during sparging.
Basic sparging approach using a collander
Wort going from the mash tun into the kettle; the rate was slowed significantly later and we spent 45 minutes getting the 28 litre final volume in to the kettle. Will aim to do this slower in futre
Karl using the refractometer, original gravity on the target 25l was slightly too high, so 28l in the kettle gave 1035 which is what we were aiming for.
Full kettle ready to boil; using the pump would avoid lifting the 28kg pot, but as mentioned thats for next time. OG was 1035 so bang on for the plan for Bwlch Mild.
50g of Fuggles hop at the start of the boil (90 minutes); very slight bittering for a Mild
Time for some guitar during the boil (as well cooking a steak) and the prepping for the cooling and getting the wort into the fermenter.
Getting the wort cool as quickly as possible using the heat exchanger. Pretty impressed with this bit of kit and we got it down to 15C in 25 minutes, with some faffing, so again as a practice brew day plenty of points for the future.
Into the fermenter, in the brewing fridge. Already set to around 20 C this should see a consistent fermentation process. Pitched the yeast starter at around 16 C, in line with the temperature of the wort at the time. The fridge uses an STC 1000 to control the fridge (to cool) and a 60W greenhouse tubeheater to heat it. Seems to work pretty well.

And now we wait for 5-7 days to check fermentation.


The devil is in the detail

An appropriate title for this website, but also relevant to the critique of a short article doing the rounds this week.

Best read the article on Red Hat’s Hidden Treasure before reading the rest of the critique below.

The article on the face of it seems spot on, as it highlights what people in Red Hat (and I assume IBM) know, that given it’s an open source company its not product or intellectual property that they are buying but the people and organisation that make it.

The biggest risks to the deal’s success might come from IBM themselves. No wonder Ms. Rometty will keep Red Hat as an independent company, despite the billions she used to buy it.

This recommendation of independence is music to the ears of Red Hat staff, most of whom who remain at Red Hat on the understanding that exactly this will happen, with IBM taking a very loose grip of the affairs of the most expensive software company acquisition in history.

However, there are initially a couple of things that make you take a closer look at the findings. The article isn’t independent, as it’s written by Avalia, a company that sells software products for assisting with mergers and acquisition, so guess what, it’s going to promote M&As in a positive light. Secondly, it mentions Docker as part of Red Hat’s product portfolio; er nope.

The article itself focuses on Ansible as an automation tool in comparison with it’s open source competitors (Puppet, Chef, Salt and Terraform) and use this as this as the basis for determining the value of IBMs acquisition of Red Hat.

It makes some good points on the health of Ansible as a product and open source community and confirms Red Hat’s approach of hedging its bets when it comes to upstream projects, having backed Puppet to a large extent previously (and incorporating it into the Satellite product). As the Avalia report states, Red Hat have chosen wisely by backing what now it is market leader in automation, Ansible and that does mean a good potential return on the investment for IBM.

There is an interesting reference at the end of the article on a wider reflection of what this specific Ansible analysis might mean to the IBM acquisition of Red Hat overall.

Taking Ansible as a reflection of Red Hat’s open-source culture, community, and technologies, IBM is making an excellent acquisition. Ansible’s core leadership belongs to Red Hat, and their care has translated into a growing and diversified community of contributors, that in turn have built a great quality software. With this acquisition, comes great power and great responsibility with Red Hat’s developer community, and many questions still need to be answered.

Avalia report

This makes a lot of assumptions, in particular about the state of the wide range of open source projects and their downstream, enterprise products that Red Hat are involved with. There are a lot of large, commercially driven open source projects like Kubernetes and OpenStack in which Red Hat plays an important part, but doesn’t have the control to the extent it does in Ansible. There are also other projects, like ManageIQ, that are primarily driven by Red Hat but are not so healthy by the same open source measurement.

Red Hat as a company also isn’t also just developers working on open source project. There is also expertise around converting this into a successful financial model based around subscriptions and services. Red Hat was the first (and maybe the only) true open source company to be significantly successful on a pure open source model.

Selling, marketing and product managing an open source commodity is very different to a proprietary one and alongside all the upstream, development goodness, people at Red Hat have also built a business. Some of this business IBM will want to leave alone, some they might mimic, but the temptation to leave alone will be harder than it would be for the developers.

A sales team working on strategic accounts, using an approach based on open source philosophies (meet-ups, customer forums, transparency through to engineering etc), will be more easily consumed by the larger acquirer. It’s likely not to be a planned approach a corporate level, but just something that happens at a lower level.

Interesting times ahead and as the Avalia article suggests, IBM are best leaving alone, but not just solely based on how they see the success of Ansible.


Peter and Jane do solutions

Peter had been working very hard on his project. Two years ago he had a new idea for a tuck shop, where instead of just buying sweets, Peter would tell people how to eat them instead.

Jane thought Peter was also working very hard too and whilst lots of people were still using the old tuck shop, Peter’s new tuck shop had really excited customers. Peter had persuaded some other children that they also liked working in the shop, because he gave them lots of sticky notes to put on the walls and windows.

Peter had the idea of having a small tuck shop on wheels so that he could tell other people to eat sweets in their own classrooms, but he didn’t do anything about it, because he was going to make more money in the main tuck shop. People would pay him lots of money so he could tell them to eat sweets properly, his way.

Jane also knew that another boy Charles had a mobile tuck shop and as well as telling people how it eat sweets, he also give them the sweets as well. Jane thought Charles had the better idea. She liked Charles more than Peter.

Peter stopped talking to Jane as he thought he had the idea first and he didn’t want to play with Charles.

The Headteacher also decided that Charles’ idea was best and told Peter if he wanted to do a mobile tuck shop he’d have to give the children the sweets once he’d told them how to eat them He’d also have to call the shop, Charles’ Sweets.

Peter was grumpy because it was his idea first. He told everyone to boycott Charles’ shop as he had the idea first.

Jane thought Peter was being a twat. After a while nobody went to Peters shop, as getting your head stuffed down the toilet by Jasper, the school bully was a more pleasant experience.


Looking both ways

He’d picked up the last clean cup from the kitchen cupboard in the office and made some tea, before realising the shape of the handle meant it was designed for right handed people. This would make drinking the murky cup of Earl Grey interesting.

Brian had worked as a technology consultant over the last 20 years and over time what had been purely keyboard-based activities had developed into more complex projects, where the process and the needs of the people using the technology had become the focus. Gone were the days were you could hide in the server room and keep your head down, happily focusing on installing software, writing scripts and submitting bugs. Increasingly he find himself in more and more meetings, discussing how the products would be used and what the customer would need to change or adapt in their way of working in order to be successful. Starting off with assessing their maturity, he would then look to encourage people and teams of develop processes and ways of working to ensure they could use the technology he would be installing.

The company Brian worked for had built a reputation for innovation and has successfully disrupted the IT market, becoming a leader in its main operating system area. As well as it’s products, it was now selling it’s brand and way of working, as well as its open organisation as something that it’s customers should emulate as part of digital transformation. It wasn’t just about the technology anymore.

This change wasn’t easy for customers and as such they looked for help in making organisational and cultural changes which would be needed to ensure their own businesses would be successful. Technology was only a small part of the jigsaw and whilst the problems were now increasingly complex, they weren’t insurmountable if approached the right way and with the right engagement.

This past few weeks had been typical of the type of organisation he was working with and Brian saw himself as much as consultant of people than he was of the technology he’d install. Some of the things he was seeing weren’t unique and all provided challenges that needed to be overcome.

  • The main software system used by the business had been migrated to over the last 12 months, with considerable effort and manual data migration. The person in charge of the transformation to the new system was no longer with the company, having taken the fall for the failures which had significant business impact. It was working but was still not providing the level of service the old system did.
  • Despite this, the main Director of the organisation was dynamic and had successfully grown the business across a number of regions; he knew that he needed to change both the technology and organisation to ensure this growth continued. He’d recently proposed a new org structure, which he’d presented to his leadership team once he’d put it together. Whilst there been some tweaks, it was his changes that were to be implemented, but the lack of detail in plan was impacting some of the key workers, who were now not clear of their role and responsibility
  • Whilst the leadership team worked together, the level of shared responsibility they had for the overall success of the business was low. Initially there had been a lot of discussion on the need to become more agile and flexible in tackling problems, but relatively quickly this had broken down into hierarchical structures with people owning resources, budgets and not cooperating
  • The central team provide systems and services to a number of Regions. These Regions had become increasingly independent and less accountable. Focused on targets there’s was an increasing diversification in products and offerings around the technology and whilst they were successful they could do things there own way. However it was becoming evident that this was beginning to be a problem with one of the largest regions failing to meet targets. These were specific to the Region and as such harder to fix by the central team
  • There are plenty of cases where individuals were propping up bad projects through hard work. These weren’t being addressed by management and people were simply left there until they went sick or the project finally crashed.
  • They’d brought in new people to help scale the systems and business as they grew, but it had infact led to significant clashes in culture. There was an increased aversion to risk, being accountable and stepping up to take on tasks. Management to individual contributor rations had gone from 10:1 to 3:1 with a significant loss in revenue.
  • A few years ago, they’d set-up and funded a small team to try out new ways of working and take on new business. It had had a measure of success and taken on new projects and won new customers, but it now saw itself as its own business rather than being part of the larger organisation. One of the main problems is that was itself looking to prevent innovation in other parts of the business, preferring to be the sole place where new ideas and technology would come from.

All of these things were typical of the challenges facing organisations who were having to transform as technology itself became fundamental for the business. As technology took centre stage, the need for effective processes and organisations become more important, not less.

Brian continued to look at the assessment and write the report he was working on. He would normally be bolstered in the knowledge that he was, as a visitor to the organisation, able take a more abstract, third party view and provide lots of useful insights to the management in his report.

However, this wasn’t the case with this report. Rather than being on customer site, Brian was benched and the organisation he was writing about was his own.

What had been the innovator and the disruptor was now struggling to find its way, with its culture eroded through expansion and the need itself to meet targets. Being on the inside and within the hierarchy now made it virtually impossible for Brian to be heard or understood. Rather than objective observation people saw personal criticism and confrontation. The more you told someone, the more they avoided you and sought out people who agreed with them and confirmed what they wanted to here.

Brian closed the laptop, looked up and out of the office window to see the snow falling gently in the car park and realised that he’d come to the office on his motorbike. Getting home was going to be a challenge with some risk, but he was up for that.


Mynnyd Troed

Last minute decision to head up my nearest mountain (that is a distinct peak over 600m). That isn’t Mynnyd Llangorse (which is only 515m) so need to get in the car and drive round to the pass between the two. It was already 14h30, so not much time for a long walk.

It’s a steep ascent, with most of the other people parked there heading up the nice looking ridge to Mynnyd Llangorse. Up to 500m and then then into the clouds.

Mynnyd Troed : not the finest view, but the best sound

It’s not the more leisurely of walks up to the trig point and the greasy ground didn’t make for easy going as I’d decided not to change out of my workshop trousers and my old walking boots. The cloud down meant there was no view; bad as there was no view but good as the view inside was much clearer.

Accompanying the tinnitus was purely silence. Cold and damp and little wind it created a little world around you. Then you catch your breath from the stiff ascent and that interrupts your thought briefly. You look around expecting to see someone else, but there is only you. It’s a magical experience that can only be experienced in the mountains and only at a certain time. Other, sunnier days, you can lie back in the grass and listen to the skylarks rising up high above you, whilst on others all that is with you is the wind.

The trig point was a limpid beacon from which a mental note of the return route was make. I’d though initially of going along the ridge, but decided against it; I could stand here for 5 minutes or so for the same effect and I’d done my workout on the 250m of ascent from the car. Slowly I made my way down, walking in the heather and rough grass to avoid ice-like short grass. The cloud didn’t clear as expected and only just before the road did the view down the u-shaped valley towards Cwmdu appeared.

Only 1 hour and 20 minutes of walking but refreshing of body and mind and whatever the weather, distance or hill (or mountain) and great diversion from the everyday.


Do technology companies really care ?

Having worked in public sector  for 10 years up to 2000 and with a US technology company since then, it’s surprising that it’s taken me so long to consider this question. Do technology companies really care ? There are lots of statements on social responsibility from them but are they more than lip service ?

This article discusses whether caring about the social impact of a project is beneficial for both end users and shareholders and if this is possible ?

Malcolm Herbert

Whether its the technology company giants (like Facebook, Google, Amazon etc), business technology providers (like VMware, Red Hat, IBM etc) or small startups, the same question should be asked, do they really have a social conscience ? As well as measuring financial performance, they may measure other key performance indicators (KPIs) and run customer satisfaction surveys, but does this show they care whether the products actually benefit the customer (or a customers customer).  Recent analyst reports and guidance and highlighting the need for social responsibility as a component for future success. 

Redefine your company based on the company you keep

Accenture Technology Vision 2018

Lets look at an example (and it’s a fictitious one). A technology company, SoftwareA, sells a products to CompanyB, that will potentially allow it develop better software, more quickly so that its customers can register complaints they may have with property rentals and landlords.  CompanyB sees regional and local government authorities as the main buyers of it’s product which links a comprehensive set of data on properties, landlords and transactions, with a nice easy to use interface, large amounts of date and some innovative software that gets information via  machine learning that enables problems to solved more effectively. The aim is that landlords fix problems quicker and that renters have less problems. 

LocalGovC buys the service from CompanyB, which includes the software, and this comes as a managed service, so that it is run and operated by CompanyB entirely. SoftwareA have provided the software that allows CompanyB to update the software and develop new functionality. 

Typically a company like SoftwareA will charge a license (or subscription) for the software product, sell some consulting to install and set it up as well as training for the developers and users of their product.  The normal measure of success for company SoftwareA is how much CompanyB spend with them, though they might also take other measurements like a Customer Satisfaction survey into account.  

CompanyB will also be deemed successful primarily on the amount that LocalGovC spends and if they are able to deliver the services at a profit.  As with SoftwareA, CompanyB should also be looking to the longer term and ensuring that their customers buy more product and renews any subscription, consulting or support services. Therefore, rather than a one-off deal, any corporate company will look to develop a longer term relationship with customers based on mutual success. 

LocalGovC will be also seen to be successful if it has bought a service under budget and that meets the requirements laid out by a wider government policy. However, this is where the pyramid of mutually assured success begins to break down. This is in part because 

However its possible that the IT based service provided by LocalGovC on its own will fail to improve the quality of social housing and the rental market in its area.  The service might have been poorly designed and not meet the needs of its customers, but to staff and management at LocalGovC this may not be the view. The metrics based on usage, response times etc might be being met, so whilst overall more tenants are suffering revenge evictions, rental prices are increasing and fewer landlords are offering affordable properties to the market, measures of success being used by LocalGovC are still being met. 

Whilst the organisations involved may see that they are responsible for the success of the final service (that is the improvement of rental accommodation on the LocalGovC area), it’s less likely the SoftwareA and CompanyB will be bought in. Things however are changing and it’s clear that to be more successful, even against the limited criteria discussed above, the software companies and service providers need to work on other areas outside of technology . However, by being more involved with process and people and not just technology, the technology companies by necessity have to care more the overall success of the project.  

Larger outsourcers haven’t got a great press and whilst they take on all the roles (including to large extent that of LocalGovC)., their drivers are that of a commercial enterprise, not of a social one and as such caring and responsibility to customers are secondary to that of shareholders. 


Kakistocracy and a thicket of idiots

A few weeks ago I was sent via facebook messenger the Wikipedia page on the Dunning-Kruger effect.   An initial read of the first couple of paragraphs and I found myself nodding my head in agreement (mainly with relationship to some work colleagues).

Further reading after laughing and further cussing, you settle into the understanding that this is actually quite a serious assessment and study and probably why Dunning and Kruger won an Ig Noble in 2000 for this work.

There are plenty of other writings on the subject, some commentary of these below, as well as my own thoughts. 

Now interestingly both the Wikipedia article and Dunning Kruger mention a quotation from Bertrand Russell, the English philosopher, mathematician and general, all round interesting person. 

In their article (with a host of coauthors) “Why the Unskilled Are Unaware: Further Explorations of (Absent) Self-Insight Among the Incompetent” there is a 1951 quote from Bertrand Russell (see below), but alas it’s not cited so you cannot understand the context in which it’s written. 

Bertrand Russel 1936
Bertrand Russell in 1936

“One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision.”

If you do a quick search on Bertrand Russell quotes, then you find another quote (usually #1 in the lists) which essentially says the sames thing.  

The whole problem with the world is that fools and fanatics are always so certain of themselves, but wiser people so full of doubts.

And if you have a quick search on BrainyQuote, then lo and behold you can find a similar quotation, again without citation from Russell (though BrainyQuotes allow you to do a citation from them, raising the whole can of worms about the search for the real truth in sources). 

The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt.

However, this is to a large extent a distraction, though if pressed on the accuracy of the quote and what it means,  you may have to say that you aren’t really sure of the exact meaning. Irrespective of the versions of the quote then it’s meaning and is clear.  That some people are not aware and are incapable of being aware of how wrong or foolish they are. 

The process of understanding and accepting this statement as fact leads onto a number of other statements that need to be considered :

  • that the only people who think that they might be stupid and overly confident are the people who are probably not; that they are wise but that they doubt that they are.
  • that it will be impossible to convince individuals that they are foolish; attempting to do so will fail because their confidence is in opposite proportion to their ability to reason and their intelligence. 
  • that mechanism of assessing the foolish and overconfident isn’t possible by those around the person, because it is also true that rather than being solely applied to individuals, it is indeed  a more powerful force in groups, where self reinforcement is backed-up by reinforcement of others. 

In hierarchical organisations it is far easier for the foolish to thrive, where lack of actual ability can be hidden from managers through effective managing-up techniques [1] and from the lack of ability of the manager themselves. Rather than a meritocracy, you have a kakistocracy , where you have government and management by the most stupid.  Given the current state of politics in parts of the world, you may also want to argue that this democracy [2]. 

 The clever and wise, attempting to confront and challenge the position and ideas of the foolish, their followers and their advocates, comes with risk. Risk of accusation of  harrying, of undermining and having a personal vendetta. It also comes at risk of livelihood, future rights of representation and in extreme circumstance, risk of life. Therefore does it make sense to do so ?  Suffering fools gladly has in modern times being primarily used in its negative context, where not suffering fools gladly has been a call to wise to get off the fence and get angry with the stupid.  This confrontational approach it could be argued is doomed to failure, as the stupid will ignore any remonstration and seek others for confirmation of their views and abilities for the equally stupid. It is therefore not wiser to suffer them gladly, to live within their group, their world and make the best of the organisation that this provides ?  By using their techniques, is not very easy to the wise to coast through life in the happy knowledge of life being easier if the stupid are happy and content with their own oblivious view. 

This happy state is however hard to achieve; the stupid will place repeated, excessive calls upon others to reinforce their overstated view of the abilities as well as making decisions that negatively impact you, the wise. Their ineffectiveness means you need to do their work for them and their overstatement of achievement means that you find yourself providing evidence to counter it, in order for your own efforts to be effective and recognised. 

Obviously a kakistocracy thrives in an environment that is doing well, where resources, climate and other inputs ensure that it is doing well, despite the input and actions of the stupid. In a world which is struggling, then the stupid cannot thrive and their actions and views do not have ingredients in which to survive.  It can therefore be argued that a long game, not a short term strategy is needed by the wise, one where that ultimately the stupid will be responsible for their own undoing, rather than this being brought about directly by others. 

Shaun of The Dead : a metaphor for surviving stupid ?

For those therefore not wishing to suffer the fools gladly (or at all) there the following might be worth considering:

  • remove the nutrients from the environment where the stupid currently exist; this could be intelligent, long suffering coworkers and subordinates
  • remove the mouthpiece and routes for pontification and therefore self-reinforcement
  • provide clear and distinct means of measurement of ability based on actions of the individual, not words 
  • ring fence the groups of stupidity and use individual measurements and apply them to the group.

Reading the above, it comes across more of a mechanism for dealing with Zombie apocalypse than it does a strategy for dealing with frustrating friends, colleagues or politicians.  This sense of inevitable doom is sometimes whats faced when dealing with an increasing hoard, that’s infecting a large majority of people around you.  Sitting there watching Shaun of the Dead and you can easily transpose this to the task of explaining the real impact of Brexit to the people who your are connected. 

So, finding yourself surrounded by a thicket of idiots, you have a number of strategies for survival, but none of these provide any guarantee of success of personal survival. 

[1] Of the articles you will find online, see managing up as personal skill to be enhanced.  For example :-

[2] A useful list of government types is provided by Wikipedia, of course. So happy was I with the discovery of kakistocracy that I donated £20 to Jimmy Wales.