about
Converge is a utility for troubleshooting builds on continuous integration servers.
It solves a common problem where the cause of job failure is difficult to determine.
This is complicated further by the fact that build jobs are usually run on a build
farm where there is no access to the build agents or in more modern envrionments when
jobs are run in ephemeral containers.
With Converge it is possible to get remote shell access to such jobs. This works
by configuring the build job to connect to a Converge server using an agent program.
The agent program can be downloaded from within the CI job using curl or wget.
Next, an end-user can connect to the Converge server, a rendez-vous server, that connects
the client and server together.
other tools
Using available existing tools such as
breakpoint in combination
with a websocket tunneling tool such as
wstunnel a similar solution can be
obtained. There are however some problems with these solutions that converge is
trying to address:
- deployment: Breakpoint uses an embedded SSH server which is a really good idea but
uses the QUIC protocol for connecting to a rendez-vous server. The rendez-vous server than
exposes a random port for every client. This make deployment on kubernetes really hard
where fixed ports must be used and QUIC is also not a widely supported protocol.
- The problem with the random ports can be solved by using wstunnel running together
with breakpoint server in a kubernetes pod, where wstunnel can forward traffic over an
extern websocket connection to the local random port that breakpoint server is listening on.
- breakpoint leaves it open on how users install the breakpoint executable (agent).
- Because of the hacky nature of this setup, it is very difficult for users to use
and troubleshoot when things go wrong.
Converve server addresses these issues in the following ways:
- Use the websocket protocol both for agents and for clients, providing a fixed port and
a supported protocol for kubernetes deploymment.
- Providing online documentation where the instructions take into account the
hostname and protocol where converge is running allowing users to cut and paste
instructions that can be used without modification. In the usage page the users
can even generate the correct agent startup commands and client connection commands
based on the type of shell they are connecting to.
- Converge server provides out of the box downloads of required software. This makes sure
client and server are always up to date. In addition a protocol version check is done.
- User-friendly error messages can be given to users in most case when things do not work
out because of
wsproxy
, an SSH proxy command that also talk to the server
to tell the user if a connection is accepted and if not why not.
- A live screen showing the current sessions that are running.
- Interactivity in the user's session with notifications about timeouts and a very
simple inactivity timmeout mechanism.
- Possibility for the user to define his own shell.
- Support for unix like bash shells and command prompt and powershell.
how it works
The steps involved are as follows:
- The agent connects to converge server. If no id is specified than a new id will
be generated. The ids specified by different agents must be unique. If the agent
specifies an id that is already in use, then a new id will be generated.
When started the agent will echo the commands to connect to it in its output.
-
Since the emmbedded SSH server in the agent will allow multiple clients to connect
to it, it wants to listen for copnnections. By default it cannot do this, it just
setup a connection to the converge server, but the converge server can in general
not connect back to it because of networking. Therefore, a multiplexing library is
used to establish multiple virtual connections over a single TCP connection.
The agent can now listen for connections from clients.
- The agent connects to the converge server using the commmand specified by the agent.
The converge server can then match the agent with the client based on the id and
the connectio at network level is established.
- The embedded SSH server now performs authentication, after successful login,
a shell is spwaned and the network connection of the user is connected to it.
The connection is practically identical to a regular terminal connection. To
achieve this some magic is used to make the shell beiieve it is connected to a
terminal.
Security
The setup is such that the connection from client (end-user) to server (agent on CI job)
is end-to-end encrypted. The Converge server itself is no more than a bitpipe which pumps
data between client and agent.
Currently converge server still supports password based login but this will be disabled.
Image two people configuring an agent with the same id where one of the agents actually
gets it and other gets a new id. Now, with a password each user can access each other's
agents. This is of course highly confusing and undesirable. Converge server already support
authorized keys but this is not yet mandatory. I is made extremely easy through the
usage page to configure this, so the additional complexity should
not be an issue.
SSH and SFTP
Both ssh and sftp are supported. Multiple shells are also allowed.
Timeouts
There is a timeout mechanism in the agent such that jobs do not hang indefinitely
waiting for a connection. This mechanism is useful to make sure build agents do not keep
build agents occupied for a long time. By default, the agent exits with status 0 when
the first client exits after logging in.
When the user touches a .hold file, the agent keeps waiting for connections even
after the last client logs out, taking into account the timeout.
The sessions have an inactivity timeout. Any keypress on the keyboard by a user
is interpreted as activity.
Remote shell usage
The agent supports a --shells command-line option by which a comma-separated
list of shells can be prepended to the default search path for shells, e.g.
--shells zsh,csh,sh
(linux) or cmd,powershell
for
windows.
The agent sets a agentdir environment variable that points to
the directory where the agent is running.
}
templ AboutTab() {
@BasePage(1) {
@About()
}
}