Overview of DevManual

I have twenty years experience “in the trenches” of software developmnet - writing commercial and open source software for cutting edge ISPs, major financial houses and not-for-profit organisations, in big offices, or on globally distributed teams. And everywhere, there is a process to making and running good software, and those processes all have a similar core.

This Dev Manual is trying to teach you what that essence of software development is, in a practical, immediately useful and downloadable fashion - my ultimate goal is provide a software team with exactly what is needed to hit the ground running.

I have (will-have) built a online business that uses the Dev manual processes and code to actually run itself. Its well meta.

Successful software is partly a technical endeavour. Get that wrong and nothing else will go right. But the hard-technical stuff is say 50% of good software - the other 50% shades into softer human skills, good and bad “management practises” and as we shall see, into national and even global politics.

I was trying to create a good structure, consistent and meaningful, like an good Technical Architect, but frankly this has turned into a much more comfortable and sprawling bazaar. There is a lesson here.

It is a good idea to read this in conjunction with it’s sister book -“The Software Mind”, which should be already in your hand - just turn the book over and start reading form the back.

The Metaphor of Building Software

Discuss “The code is the design”, and the DevOps idea of code for everything. Look at building site in City of London

The office building metahpr is a good one because it also includes a clear idea of just how much is ocvered in software build chains these days - from Steel girders and foundations to Glass UIs and bathroom and waste services in conduits no one sees, to building security and power outlets. These things are beasts, and software at even mid size enterprise scale is a beast too. Hence my focus on governance as well.

Architecture

Architecture is ... hard. And very very very often misunderstood.

Most simply it is the balance between form and function.

There is by now a rough consensus on how to design a sheef of internal enterprise applications. It is basically

Microservices owned by Small Teams, with Strong Interfaces

The problem with this is of course not that it is a technical change. It is not. It is a business and organisational change. And so that means mostly it gets broken against the internal politics of companies, leading to things like huge DevOps departments and large Ops teams.

See the Software Mind chapter.

Layers of architecture ? However algorithms affect architecture -see the fractal indexes for caches

Simplicity

Our golden goal is to keep things simple.

Simple breaks in simple ways, simple is simple to extend and improve.

Its not that simple is easy - often it is the opposite of easy, or quick.

But simple wins out over time. Simple gives great ROI.

I throughly recommend listening to Rich Hickey on this subject (Link)

So please keep in mind - we aim for simple. Even if our day to day work pushes us to quick and easy and complicated. We need to push back.

Plumbing

Every software project of any size needs some basic plumbing, things like a good config approach, a means of linting and testing. These things pay dividends throughout the lifetime of a project, making the simple easy and the hard doable. WIthout it, you are in trouble.

Continuous Integration (CI)

Re-create your company from scratch, every single day.

Continuous Integration / Delivery is probably the biggest boon to developer produtivity since the rise of memory managed languages in the 90’s.

Languages like C expected the developer to write code that assigned a certain amount of memory for a certain data structure - which meant at the point of writing your code, you had to know how big the data was going to be, in say two years time.

People would get this wrong. The biggest security risk for many years was your program accepting a piece of data larger than expected and simply overwriting its own memory. With luck your program just crashed. Otherwise the hacker was very good, and the piece of data sent just put their evil code on top of the stack ready to be run.

Nowadays, the business logic we grind out does not need to worry about such things. Instead we have the fun of not being entirely sure if the version of the code we think of is going to run on the server with the code we think of, using the password we meant and thats if no one else changed anything.

Lets call these build services.

Software Governance

There is a software rule of thumb - that code-bases pass through ‘complexity horizons’ every couple of orders of magnitude. That is a project that was easy to manage at 1,000 LOC cannot be maintained with the same approach when its a 10,000 or 100,000 LOC cadebase.

Pieter Levels is a entrpreneur and coder, who found a certain noteriety in 2017 by announcing that he was making sales of over $2,000 a day, using a single php file with 4000 LOC, with no frameworks and libraries. The Twitter-sphere exploded, quite amusingly, by criticising his coding approach and insisting he needed some architecture and frameworks.

“What about the frameworks. Think of the frameworks”

Yes, at some point the ‘one guy opens up one file’ approach is of course going to fail.

How we manage that is software governance. The goal of software governance is to raise the floor everywhere.

We can write code, we can write code that gets us to a basic level of feature complete-ness. And then the next fire alarm arrives, the next email from the boss, and ... the polish disappears, the extra bit of effort to make something long term useful just does not get done.

But that extra piece of effort can pay dividends just for one developer. For a team or a whole community, the dividends are endless, just by raising the floor of quality.

In todo-inator I have a concept of self-rating each module or function with a modern form of P.G. Wodehouse’s re-writing of chapters. This simple mark:

pgw: **

While this is a subjective measure from the developer, it is a guide to where improvements can be made. And importantly resides in the codebase.

Other measures of code quality can be autoated and should be part of every commit cycle.

The Single Best Method to raise Software Quality

Code base governance

Style, coverage ast based syntax checking use of non-standard plumbing Code as a crime scene Static Analysis and raising all boats. ast and how to do syntax checking like pyflake - how to build own rules

Systems governance

Governance (dev to prod access etc) (As infrastructure as code increases, this sort of thing is more possible) this is monitoring running systems. Things like approvals, security etc.

Mission statements https://www.amazon.jobs/principles

Skills for individual developer

Software Governance as a force multiplier implies a number of things One is that each individual contributor should have the same minimal set of skills, and perform those common skills in a similar fashion.

An obvious example might be making good source code commits, and so there would need to be an internal “standard” for commits.

This of course implies ... training. Training your staff to be better at their jobs, something that the commitment-less culture these days seems to mitigate against. Things will change - our “principle” of a change to Roald Coase’s equilibrium point means smaller companies, and greater need to standard interfaces and so more need to train your people to do it the right way.

Profesional Stuff you should know

Actually personal stuff

  • chapters/interviews_algorithms

Misc - <no title>

Security

Security is principles that are applied across the system. PKI, etc.

Basically trust the maths, and trust nonces.

  • GPG and keypairs
  • host based security, networks of trust between hosts, and DMZs
  • Kubernetes / Rancher as a host / VM world

Testing - a heresy

Rick Hickey on simple vs easy

How does a bug get into production? It is written And it passes the tests. So if you have tests, and you refactor, how do you prevent that bug? Need to be able to reason about code. Which is why 900 npm packages worry me.

Tests are regression tests. They are written so that having written some code to do a thing, you dont later on screw it up and it stops doing that thing. Tests are almost by defintion, backwards looking.

DevOps

Falls naturally out of Microservices owned by Small Teams, with Strong Interfaces SRE and SRE book. Start small, keep whole thing in overview Use graphite, and just report out, graph 10 important things to your team today.

Docker AWS

We shall build a complete enterprise service in the cloud - because we can

Basic Management Reporting

  • reportlib
  • SLAs and KPIs - keeping ourselves honest
  • focusing upwards to higher levels of leverage
  • avoiding the drumbeat of deadlines, and panic, and agreeing goals based on data / 20% most effective things to fix.
  • Make one weekly report today

Soft Skills

Esprit d’corp and Team honesty

Hiring practises - be part of the team Entry hurdles. start with feedback - sprints and retrospectives Be aware of your priviledge Begin the difficult conversations publically be aware of the likely problems - metoo is just one.

then aim for the culture you want -

then hire good people

  • Culture, and hostile cultures
  • trust, safe space, I dont know
  • learning
  • lunch
  • Keep on in good faith
  • Google HR managemenet
  • management fixes are the middle ground -

Business and Software

  • serverless is cheaper. Please rewrite everything now.
  • Overtime is bad
  • remote working is more productive
  • Risk management beats project Management

Project and Programme management

It if ain’t got a ticket dont work on it If it ain’t possible to rollup tickets you dont know where you are going A backlog out of context is just a horror There is nothing wrong with top-down design (side??) Backlog for the whole company

CTO dashboards and Business Process Dashboards

Dashboards matter The basics of code quality can be in dashboard. The basics of production health can be in dashboard Putting a business process into dashboard is powerful - use Graphite and “light beam trackers”

The dev manual - a proof of concept

This is a “business in a box” - it kind of does not matter what the buisness is, its just that all the software engineering goodness that I describe here needs to be ... dmeonstrable - so I have built a example business (and launching a real product) with it.

Its WIP

  • simplest app possible
  • adding a unit test
  • adding a performance test
  • building it under python / distutils
  • running it under systemd
  • running dual, behind load balancer, using weaver/ansible/fabric
  • building it on a build server, using .deb files
  • build assets -> docs, perf results, test results, .deb files
  • Security on microservice
  • Identity
  • host-host services (ntp etc)
  • host-app services -> logging, TLS etc
  • central services - DNS, metric names,
  • code reviews and code promotion
  • metrics gatehrinfg
  • log mgmt
  • rolling out changes
  • incident mgmt (incidents, SLAs, uptime measurements from metrics etc etc)
  • adding message queues, backend services, passing back identiy
  • adding dependancy services - monitoring everything
  • CTO dashboard, mission control centre
  • bug tracking, feature development

Putting it all together

  • Simplest possible We shall build a working web app (about three lines, honest). Build it, test it, deploy it to a location locally, and log it.
  • systemd, well-behaved services
  • simplest app possible
  • adding a unit test
  • adding a performance test
  • building it under python / distutils
  • running it under systemd
  • running dual, behind load balancer, using weaver/ansible/fabric
  • building it on a build server, using .deb files
  • build assets -> docs, perf results, test results, .deb files
  • Security on microservice
  • linting and style and code reviews
  • Identity
  • host-host services (ntp etc)
  • host-app services -> logging, TLS etc
  • central services - DNS, metric names,
  • code reviews and code promotion
  • metrics gatehrinfg
  • log mgmt
  • rolling out changes
  • adding message queues, backend services, passing back identiy
  • adding dependancy services - monitoring everything
  • CTO dashboard, mission control centre
  • bug tracking, feature development
  • distributed file systems Cephfs, GlusterFS, Lustre, and HDFS
  • work queues CElery, zeroMQ
  • amazon, openstack

Micro-HowTos

(Misc)

The top 12 practices - a summary

Like Joel’s checklist, this is a checklist for things you need Its trying to get ot barebones

  1. source control 5 chars etc. but good example of using automated policy enforcement on checkin
  2. tech debt and tech assets - code and tests
  3. requirements lifecycle (PEP) the wrongest part of the agile manifesto “”” The most efficient and effective method of

conveying information to and within a development team is face-to-face conversation.

“”“

Ya do need to write down the discussion. written Proof overcomes authority problems

it is also way to get everyone discussing this only works with really co-locateed and mission focused teams
  1. automated build and deployment (dogfood) Look, bash is just fine pyholodeck
  2. Documentation and Marketing
  3. openness and reviews
  4. Progress Not Perfection (YouTube clip)
  5. static and other analysis
  6. performance mgmt and measuring everything (and making reports on everything)
  7. Automatic project mgmt
  8. Risk management
  9. have fun, try new things, don’t be afraid