Comprehensive set of protocol versions.

[evoskuil]

Hi, we are close to completion on the libbitcoin-server electrum interface. This is one of several client-server interfaces to the node, which optionally populates the address index while building the chain. We've provided what we believe is a full implementation for the following versions, based on available documentation. Since we only support explicit version matching, we want to make sure that we've got the versions covered. We aren't implementing any altcoin extensions, just btc mainnet/testnet. Could you let me know if this is complete? Thanks

enum class version
{
    /// Invalid version.
    v0_0,

    /// 2011, initial protocol negotiation (unsupported).
    v0_6,

    /// 2012, enhanced protocol negotiation (unsupported).
    v0_8,

    /// 2012, added pruning limits and transport indicators (unsupported).
    v0_9,

    /// 2013, baseline for core methods in official specification (unsupported).
    v0_10,

    /// 2014, deprecations of utxo and block number methods (minimum).
    v1_0,

    /// 2015, updated version response and introduced scripthash methods.
    v1_1,

    /// 2017, added optional parameters for transactions and headers.
    v1_2,

    /// 2018, defaulted raw headers and introduced new block methods.
    v1_3,

    /// 2019, removed deserialized headers and added merkle proof features.
    v1_4,

    /// 2019, modifications for auxiliary proof-of-work handling.
    v1_4_1,

    /// 2020, added scripthash unsubscribe functionality.
    v1_4_2,

    /// There is no v1.5 release (skipped).
    //v1_5,

    /// 2022, updated response formats, added fee estimation modes (maximum).
    v1_6,

    /// Not yet valid, just defined for out of bounds testing.
    v1_7
};

    static const std::unordered_map<version, std::string_view> map
    {
        { version::v0_0,   "0.0" },
        { version::v0_6,   "0.6" },
        { version::v0_8,   "0.8" },
        { version::v0_9,   "0.9" },
        { version::v0_10,  "0.10" },
        { version::v1_0,   "1.0" },
        { version::v1_1,   "1.1" },
        { version::v1_2,   "1.2" },
        { version::v1_3,   "1.3" },
        { version::v1_4,   "1.4" },
        { version::v1_4_1, "1.4.1" },
        { version::v1_4_2, "1.4.2" },
        { version::v1_6,   "1.6" },
        { version::v1_7,   "1.7" }
    };

[SomberNight]

Hi. Nice!

Do you really want to implement support for protocols older than, say, 1.0 (~2015-06)?
In practice, I think ~no one would complain if you only implemented 1.4 (~2018-07) and newer. Note that ElectrumX (server) has only supported serving clients with protocol 1.4+ since 2019-04. There is no modern client out there that wants an older version.

Anyway, for protocol versions newer than 1.0, this present git repo is a good source.
https://github.com/spesmilo/electrum-protocol/blob/master/docs/protocol-changes.rst
https://electrum-protocol.readthedocs.io/en/latest/protocol-changes.html

For versions older than 1.0, you can look at the Electrum client source and original electrum-server source.
Based on git history of electrum-server src/version.py, versions {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10} all existed. but IIRC you are right that the client did not care until around 0.5 or 0.6.
If you look at the client code, it only mentions {0.5, 0.6, 0.9, 0.10}.

The important part, the 1.0 and newer versions you list, looks complete.

edit: oh, I just noticed in your comments you mention that the old versions are unsupported, well nvm then :)

---

Could you clarify what exactly you mean by "we only support explicit version matching" ? The client in server.version is allowed to send a range of protocol versions, for the negotiation, do you support that?

[evoskuil]

Thanks for the info @SomberNight !

By explicit version we mean that we only support the listed specific versions, not for example a continuous range of integer values. IOW a version of "1.5" would be considered invalid and the handshake would fail. We do support both a single and range of version numbers. The versions are ordered via the enumeration, so as long as the upper and lower bound are valid versions, the range match will work as expected.

We do tend to support full history when it comes to protocols, as we are designed as both a developer toolkit and historical/educational reference but in the interest of time I think we'll cut this off at 1.0.

[evoskuil]

This change history was helpful, thanks. We were previously operating from documentation which is missing some of that detail.

Could you clarify that v1.4.1 changes are limited to altcoins, but that some instance of btc wallet sends this version? And that v1.4.3 is also for altcoins, but that no instance of the btc wallet sends this version?

We were missing the addition, default change, and removal, of the headers.subscribe raw parameter. Could you point me to documentation on the non-raw behavior (v1.0-v1.1)?

We were missing the first three changes to the version/handshake behavior. These raise three questions:

• In the case of < v1.6 what is the expected behavior when a message is sent prior to the server.version message? What version does the server assume for the response?

• In the case of multiple version messages, is the version renegotiated and updated, or is this part ignored?

• What specifically are the semantics and return value changes from v1.0 to v1.1?

[SomberNight]

Old versions (<1.4) have really weird quirks, IMO it's more work than its worth to try to be compatible with them.

In general, for old protocol version docs (1.0-1.4), start here:
https://github.com/spesmilo/electrumx/blame/1.15.0/docs/protocol-methods.rst
and git blame

---

The versions are ordered via the enumeration, so as long as the upper and lower bound are valid versions, the range match will work as expected.

So is it the case that if a client connects and sends e.g. server.version("fancy client 123", ["1.4", "1.9"]) then you(server) error/disconnect, as protocol version "1.9" is not a known version? That sounds overly restrictive and contrary to the spirit of sending a range. That would make rolling out new protocol version even more problematic: clients could not even ask for new versions until they expect the server to support it -- or the client would have to try connecting multiple times, trying to negotiate different protocol versions each time.

The version negotiation is not perfect but it's mostly problematic for the client. From a server's PoV, it's fine I think.
ref: #6 (comment)

By design of server.version, clients must support a continuous range of versions.
OTOH, servers are free to pick-and-choose any particular protocol version they want and can skip interim versions.

Could you clarify that v1.4.1 changes are limited to altcoins, but that some instance of btc wallet sends this version? And that v1.4.3 is also for altcoins, but that no instance of the btc wallet sends this version?

You are right I think, neither 1.4.1 nor 1.4.3 is interesting to Bitcoin. But I am not sure whether any non-altcoin client might send those versions. You could map 1.4.1 to 1.4, and 1.4.3 to 1.4.2.

We were missing the addition, default change, and removal, of the headers.subscribe raw parameter. Could you point me to documentation on the non-raw behavior (v1.0-v1.1)?

see e.g. spesmilo/electrumx@47f65ff

In the case of < v1.6 what is the expected behavior when a message is sent prior to the server.version message? What version does the server assume for the response?

Right... that was badly designed. I don't think there is a good way to handle it in general.

see SomberNight@8e26c35 :

When a connection is made, both client and server must initially
assume the protocol to use is their own protocol_min.

The client should send a :func:server.version RPC call as early as
possible in order to negotiate the precise protocol version; see its
description for more detail. All responses received in the stream
from and including the server's response to this call will use its
negotiated protocol version.

see weird stuff old electrum clients were doing:
https://github.com/spesmilo/electrum/blob/82e88cb89df35288b80dfdbe071da74247351251/lib/network.py#L305-L320
spesmilo/electrum@35878b4
spesmilo/electrum@e0a6b08

I think if you default the protocol version to 1.0 until it is negotiated, that should work for all old clients I remember from the 1.0-1.4 era.

In the case of multiple version messages, is the version renegotiated and updated, or is this part ignored?

see spesmilo/electrumx@5ba5d05
AFAICT older ElectrumX used to update the used protocol version on-the-fly (so allowed re-negotiation of the version).
If you want to keep your sanity, I would suggest erroring out if a client tries to renegotiate AND change the protocol version for an existing session. While old electrum clients used to send server.version many times in a session (it was used as ping for keep-alive), they always asked for the same protocol version. I would be surprised if there was any client out there that tried to change the protocol version multiple times.

version/handshake behavior
What specifically are the semantics and return value changes from v1.0 to v1.1?

spesmilo/electrumx@9eee1c8
https://github.com/spesmilo/electrumx/blob/70dfaa895de986f2466e1d0b699294725e69bcf8/docs/protocol-methods.rst?plain=1#L838-L846

https://github.com/spesmilo/electrumx/blob/70dfaa895de986f2466e1d0b699294725e69bcf8/server/session.py#L360-L364
https://github.com/spesmilo/electrumx/blob/70dfaa895de986f2466e1d0b699294725e69bcf8/server/version.py#L3

The client used to send its name and a single protocol version, the server selected that protocol version or errored, and then the server just returned its name and version, e.g. "ElectrumX 1.2.1".

[evoskuil]

Thanks, this is very helpful. You're right about the first comment. We'll change that.

The 1.0-1.4 era is already complete, and we like to hold full history, as a sort of academic/historical target. We do the same for btc consensus and p2p versioning (though still working backwards on some of that).

We can accommodate the above versioning idiosyncrasies. Even on-the-fly is not a problem, because subscriptions across the three address types are consolidated, and that's the only statefulness apart from the version info itself.