Thursday 22 February 2024

REMEMBRANCE.md - Tombstones for git projects

There is something wrong with the idea that git gives you confidence to delete.

As so often this is from the perspective of the individual programmer.

Sure, you can delete your code and you can retreive it. Well done you, but you are relying on your memory.

Unlike mere programmers Developers are fully interchangeable members of a multi-decade team.

Tombstones

In the world of online publishing there is a concept of a 'tombstone' which is located at the URL of a retracted paper or any other entity removed for a reason. There are similar usages in programming.

A Book of Remembrance as a collection of Tombstones

My grandfather's ashes were, I believe, scattered at sea, but as a child I was impressed that he still had an entry in the Book of Remembrance at the Crematorium, which gave us a place to visit, and as my unsentimental Grandmother might have said, took up a lot less room.

Now these remebrances are virtual.

Gone but Not Forgotten

Even after using git for a long time Developers still do not delete modules, files or even lines of code. They comment them out, or just leave them in place, holding an enormous amount of context in their heads.

Let's not criticise them, lets take their concern seriously. They want to be able to find things themselves and for future Developers to know what was useful, even if it is not currently required.

REMEMBRANCE.md

At the top of every repository there HAS to be a README.md (or you get nagged by github); similarly we should add a REMEMBRANCE.md which takes the following form:


| File                |    Date    | Description and Reason for Deletion                          |
|---------------------|:----------:|:-------------------------------------------------------------|
| maven-tomcat-plugin | 2013-11-11 | An embedded Tomcat, for testing. The team just walked away   |
| my-perl-script.pl   | 2019-03-23 | A script to remove stones from horses hooves. Moving on up.  |
| tpp-module          | 2024-02-22 | Finished polishing, flushing now.                            |

Lest We Forget

Example REMEMBRANCE.md
FileDateDescription and Reason for Deletion
maven-xxx-plugin 2013-11-11 An embedded Tomcat, for testing. The team just walked away
my-perl-script.pl 2019-03-23 A script to remove stones from horses hooves. Moving on up.
tpp-module 2024-02-22 Finished polishing, flushing now.

This simple practice will give Developers the confidence to delete and unleash the promise of clean projects that git offers.

PS Spelling

It is Remembrance, not Rememberance, I looked it up, though I prefer Rememberance.

Friday 16 February 2024

Melati Day - Twenty Five Year Anniversary

Happy Melati Day!

On February 16th 2000 WilliamC made the first commit to Melati.

https://github.com/Melati/Melati/commit/7a6fb6762cdbb835b4ff1aeb987bf1671cc598bd

The name ...

  Indonesia's national flowers: the melati (small white sweet smelling
flower) ...
:-)
Why it's a coffee variety (at my favourite shop here in The Hague) I
don't know---I'll ask her next time I'm in.

On 11th July that year we held Melati One.

PS Some five years later git was born!

Thursday 25 January 2024

Five Element Maven Version Numbering System (N5)

Five element unique

The Five Element Maven Version Numbering System (N5) addresses the problem of wack-a-mole Maven dependency management: through the combination of excessive modularisation, microservices fervour, XML Declarative Project Object Models and non-unique (SemVer) versioning enterprise Maven Java projects become unmanageable after about five years.

The solution is a sequential identifier which is unique across the space, and does not require xml parsing to modify.

My first three proposals represent blundering steps towards the N5 Maven Version numbering system.

N5 Unique Versioned Artifact Identifier (Maven version)

\dCode for project
\d\d\dCode for repository
\d+Major
\d+Minor
\d+Patch

The previous version, N6, included the Year between the Repository Code and the SemVer Major Version. This was considered too much.

2.130.2.0.6 works well with the Maven tooling and can be updated over the whole space using sed.

Friday 19 January 2024

Maven Version Versions and a Regex Block Refactoring of Sequential Identifier

In the first post in this series I had thought about the need for a globally unique incrementable artifect id, in the next post I not only designed but tested a proposal for such a globally unique artifact id and how to change it.

A triumph of thought and testing over trial and error.

So I tried using the new scheme with the Maven Release Plugin.

A depressing journey into the rabbit hole that is Apache, Maven and the Release Plugin, a crucial tool in our pipeline.

It turns out that whilst the Maven Artifact Version I had tested against allowed my new scheme the Release Plugin has its own subset of such versions which are gate-kept by a Regular Expression.

          "^((?:\\d+\\.)*\\d+)" // digit(s) and '.' repeated followed by digit (version digits 1.22.0, etc)
                    + "([-_])?" // optional - or _ (annotation separator)
                    + "([a-zA-Z]*)" // alpha characters (looking for annotation - alpha, beta, RC, etc.)
                    + "([-_])?" // optional - or _ (annotation revision separator)
                    + "(\\d*)" // digits (any digits after rc or beta is an annotation revision)
                    + "(?:([-_])?(.*?))?$"); // - or _ followed everything else (build specifier)

This stops any version which does not start with digits and full stops from being recognised as a version.

Current Incremented Outcome
VTW360-2023-1.0.3-SNAPSHOT Unable to parse the version string F
VTW360.2023.1.0.4-SNAPSHOT Unable to parse the version string F
VTW360-2023.1.0.4-SNAPSHOT Unable to parse the version string F
VTW-360-2023.1.0.14-SNAPSHOT Unable to parse the version string F
VTW-360.2023.1.0.16-SNAPSHOT Unable to parse the version string F
2023-VTW360-1.0.3-SNAPSHOT 2023-VTW361-1.0.4-SNAPSHOT Eh?
2023|VTW-360|1.0.14 Illegal characters in jar name F
2023#VTW-360#1.0.14-SNAPSHOT 2024#VTW-360#1.0.14-SNAPSHOT Eh?
2023.VTW-360.1.0.14-SNAPSHOT 2024.VTW-360.1.0.14-SNAPSHOT Eh?
360.2023-1.0.15-SNAPSHOT 360.2023-2.0.15-SNAPSHOT Eh?
2023.360.1.0.14-SNAPSHOT 2023.360.1.0.15-SNAPSHOT Meh
1.0.19-VTW360-2023.SNAPSHOT hangs Eh?
1.0.19-VTW360-2023-SNAPSHOT 1.0.19-VTW361-2023-SNAPSHOT Eh?
1.0.19-2023-VTW360-SNAPSHOT 1.0.19-2024-VTW360-SNAPSHOT Eh?
1.0.19-YY24-VTW360-SNAPSHOT 1.0.19-YY25-VTW360-SNAPSHOT F
1-0-19-2025-VTW360-SNAPSHOT 1-1-19-2025-VTW360-SNAPSHOT Eh?
360.2023.1.0.16-SNAPSHOT 360.2023.1.0.17-SNAPSHOT MKay

Proposal 3

So we are forced to allocate meaning to digits and place, never a good idea, but it is that or try and change the behaviour of the Release Plugin, a task I do not think I have the time or the energy for.

\dcode for project
\d\d\dcode for repository
\d\d\d\dyear
\d+Major
\d+Minor
\d+Patch

I added the new proposal to the tests, and wrote some more tests based upon the above regex.

We land with 2.130.2024.2.0.6.

However much you think about the problem previous engineers will have constrained your choices.

Sunday 14 January 2024

Maven version schema change

code version schema change - hotpot.ai

I proposed a new versioning scheme to yield a unique version string for each artifact in our highly modularised Maven java application.

Currently we are running a classic SemVer semantic verioning system. The only shortcoming I can see with SemVer is that it is not globally unique. In the Maven world the version is combined with the artifactId and the groupId to create a globally unique token, but because this is within an XML based Project Object Model (POM) file it is fairly difficult to get that token out without buying into the whole toolchain. As Maven allows ranges (don't use ranges) and inheritance it is practically impossible for mortals to parse a pom correctly.

What problem are we trying to solve?

We want to be able to edit all poms across our collection with a single command to ensure that each version is completely deployed and we have no version clash. We have developed a tool to make the Maven Dependency Tree readable which has allowed us to see how often the team make a change but do not fully roll it out.

The problem we are trying to solve is how to update all usages of an artifact every time it is updated, ensuring we never run with a mixture of versions (almost like a Monolith).

In the wider open source community it is not the responsibility of the library writer to ensure that every usage is updated to the latest version but within an enterprise it is.

Initial proposal

My first proposal, prepending the semantic version number with a unique code for the library, looked good:

SMD110-3.0.8

This system would work if we had started with this or if we were to backfill all previous versions, however we are proposing a change only to new versions and we have to ensure that the new versioing system will play nicely with the existing versions.

By play nicely I mean that SMD110-3.0.8 should be understood to be more recent than 3.0.7.

I had planned to test this by trial and error but then I thought to actually write a test.

Testing the proposal

I found the Maven verson comparison code and wrote some tests.

The tests revealed my assumption to be incorrect (whether this is a bug or not I am not sure), but the tests enabled me to propose a schema change which did play nicely:

    @Test
    public void workableNextYearNewOldComparisonTest() {
        ComparableVersion it = new ComparableVersion("2025-SMD130-2.0.7");
        ComparableVersion previous = new ComparableVersion("2024-SMD130-2.0.6");
        assertTrue(it.compareTo(previous) > 0);
        assertTrue(previous.compareTo(previous) == 0);
        assertTrue(previous.compareTo(it) < 0);
    }

Revised proposal

Prepending the initial proposal with the year is an improvement, the year of publication is useful and it works with the tooling:

<year>-<artefact code>-<semantic version>

2024-SMD110-3.0.8

Test your assumptions, it is easier than trial and error!

Thursday 30 November 2023

Unique versions as pitons out of Maven Dependency Hell

The whole Maven ecosystem, and more, follows the Semantic Versioning proposal from @mojombo who made a success of being right about most things.

Even Oracle use it (no more ojdbc.jar for them).

The problem Semantic Versioning addressed was the opposing forces of

version lock
the inability to upgrade a package without having to release new versions of every dependent package
version promiscuity
assuming compatibility with more future versions than is reasonable
leading to Dependency Hell.

If that is the problem you have then Semantic Versioning will help. Semantic Versioning helps you live within the Maven ecosystem, if you are contributing to that ecosystem then use Semantic Versioning.

That is not the problem that you have if you dug your very own Dependency Hell due to having a modularisation addiction.
If the problem you have given yourself is creating over a hundred inter-dependent modules because of your desire to obey the injunction Do not Repeat Yourself (DRY) within a modular microservice world the problem you have is achieving consistency.

The Semantic Version string MAJOR.MINOR.PATCH is not unique.

What is actually unique is ARTIFACT.MAJOR.MINOR.PATCH.
Or even uniquer (sic) GROUP.ARTIFACT.MAJOR.MINOR.PATCH.

This is given in the Maven Project Object Model (POM) eg

  <groupId>org.melati</groupId>
  <artifactId>melati-parent</artifactId>
  <version>0.8.0-SNAPSHOT</version>
Using Maven and Java and yadayada you can parse this and get the actual unique identifier you want, but life is too short.

We want to use line based unix tools, grep and sed.

Typical Semantic Version strings

3.0.7
6.0.8
2.0.5
2.0.1
5.0.4
2.0.2
1.0.2
1.0.2
2.0.7
1.1.16
3.0.1
1.0.2
1.0.2
5.0.1
5.0.1
5.0.2
5.0.1

These are not all unique and those that are are accidentally unique.

Another way of achieving the same objective would be to duplicate the artifactId in the version. <version>melati-13.0.15-SNAPSHOT</version>

Or to merely ban whitespace: <artifactId>melati</artifactId>
<version>13.0.15-SNAPSHOT</version>
becomes <artifactId>melati</artifactId><version>13.0.15-SNAPSHOT</version>

This is fragile: someone will reformat the POM using their IDE and it will fall apart.

To achieve uniqueness we will prepend with an Artifact Code, this code will retain the alphabetic ordering of the artifactId and be gappy so that we can insert new artifacts without breaking that ordering. To make it easy to separate from the Semantic Version the code will be followed by a hyphen not a full stop.

SMD110-3.0.8
SMD120-6.0.9
SMD130-2.0.6
SMD140-2.0.2
SMD150-5.0.5
SMD160-2.0.3
SMD170-1.0.3
SMD180-1.0.3
SMD190-2.0.8
SMD200-1.1.17
SMD210-3.0.2
SMD220-1.0.3
SMD230-1.0.3
SMD240-5.0.2
SMD250-5.0.1
SMD260-5.0.3
SMD270-5.0.2

Now that we have a unique string we have pitons to climb out of the hell we dug for ourselves.

	find * -name pom.xml | \
	grep -v target | \
	xargs sed -b --in-place  's/old-unique-version/new-unique-version/g'

Thursday 28 September 2023

Linux Laptop 2023 as a present

My eldest has been using Linux on a 2012 macbook pro with 16GB and an SSD where the optical drive used to be, something she did herself whilst at school, so for her twenty first birthday I wanted to give her a new Linux laptop which she could be proud of and which also had my stamp on it.

I like Linux, OpenSource and plucky engineering, she likes sustainability and being able to run python simulations.

The problem with the macBook is that it feels like the wifi card is beginning to fail, the screen is cracked and it is heavy by today's expectations.

Canvassing the opinion of the remnants of the Oxford Linux Users Group and googling gave me the following candidates.

Candidates

XPS 13 Plus Laptop

https://www.dell.com/en-uk/shop/cty/pdp/spd/xps-13-9320-laptop/

A phenominal choice of screens, but a maximum of 32mb memory. A three year old Dell laptop would be "pocket money prices", as will this be in three years. Work will give you a Dell. Not a twenty first birthday present.

System76 Pangolin

https://system76.com/laptops/pangolin

Not sure about the logo. Does not seem to believe the UK exists. 64GB not available. Shipping costs are high.

Framework Laptop 13

https://frame.work/gb/en/products/laptop-diy-13-gen-amd

Runner up. Nice idea but I am not convinced anyone wants this in a laptop.

Starlabs Starbook

https://starlabs.systems/products/starbook

Blue Cables; 64gb ram; pre-installed Ubuntu; British startup.

Winner