Converge is a utility for troubleshooting builds on continuous integration servers. It solves a common problem where the cause of job failure is difficult to determine. This is complicated further by the fact that build jobs are usually run on a build farm where there is no access to the build agents or in more modern envrionments when jobs are run in ephemeral containers.
With Converge it is possible to get remote shell access to such jobs. This works by configuring the build job to connect to a Converge server using an agent program. The agent program can be downloaded from within the CI job using curl or wget. Next, an end-user can connect to the Converge server, a rendez-vous server, that connects the client and server together.
The setup is such that the connection from client (end-user) to server (agent on CI job) is end-to-end encrypted. The Converge server itself is no more than a bitpipe which pumps data between client and agent.
Both ssh and sftp are supported. Multiple shells are also allowed.
There is a timeout mechanism in the agent such that jobs do not hang indefinitely waiting for a connection. This mechanism is useful to make sure build agents do not keep build agents occupied for a long time. By default, the agent exits with status 0 when the first client exits after logging in. This behavior as well as general expiry can be controlled from within a shell session by touching a .hold file. After logging in, the user can control expiry of the session as instructed by messages in the ssh session. When the timeout of a session is near the user is informed about this with messages in the shell.
In a continous integration job, download the agent, chmod it and run it.
# linux
curl http{{.secure}}://{{.host}}/docs/agent > agent
chmod 755 agent
./agent -id ID ws{{.secure}}://{{.host}}
# windows
curl http{{.secure}}://{{.host}}/docs/agent.exe > agent.exe
agent -id ID ws{{.secure}}://{{.host}}
Above, ID is a unique id for the job, the so-called rendez-cous ID. This should not conflict with IDs used by other agents. The ID is used for a rendez-vous between the end-user on a local system and the continuous integration job running on a build agent. The agent to the converge server and tells it the ID. Clients can now connect to the Converge server to establish a connection to the CI job through converge by also specifying the same ID. Communication between end-user and agent is encrypted using SSH and the rendez-vous server is unable to read the contents. The rendez-vous server is nothing more then a glorified bit pipe, simply transferring data between end-user SSH client and the agent which runs an embedded SSH server.
NOTE: When running the agent on windows, an exit of the remote session using exit in powershell or command prompt does not terminate the shell completely. To terminate the client ssh session must be killed by closing the terminal. Cleanup of remote processes on the agent appears to work properly despite this problem. It is just that exit of the windows shell (powershell or command prompt) is not detected properly.
The agent has more options, download the agent and run it without arguments to see all options.
wsproxy is a command that can be used as a proxy command for SSH which performs the connection to the
remote server. This command needs to be downloaded only once (see downloads below). It does not depend on
the converge implementation but only on the websocket standards. Other tools that
provide a mapping of stdio to a websocket can also be used instead of wsproxy.
Next step is to run a local SSH or SFTP client:
ssh -oServerAliveInterval=10 -oProxyCommand="wsproxy ws{{.secure}}://{{.host}}/client/ID" abc@localhost
sftp -oServerAliveInterval=10 -oProxyCommand="wsproxy ws{{.secure}}://{{.host}}/client/ID" abc@localhost
This option is less convenient than the proxy command because it requires two separate commands to execute.
Local clients can connect using regular ssh and sftp commands through a tunnel that translates a local TCP port to a websocket connection in converge. See the downloads section. This runs a local client that allows SSH to port 10000 and connects to converge using a websocket connection.
Next step is to run a local SSH of SFTP client:
ssh -oServerAliveInterval=10 -p 10000 abc@localhost
sftp -oServerAliveInterval=10 -oPort=10000 abc@localhost
The abc user above is defined by the Converge server and
communicated to the agent when the agent is started.
Another way to authenticate is through an .authorized_keys file in the
same directory as where the agent is started.
This can be setup as follows before starting the agent:
# linux
echo "ssh-rsa dkddkdkkk a@b.c" > .authorized_keys
echo "ssh-rsa adfadjfdf d@e.f" >> .authorized_keys
# windows
echo ssh-rsa dkddkdkkk a@b.c > .authorized_keys
echo ssh-rsa adfadjfdf d@e.f >> .authorized_keys
Note that on windows you should not used quotes.
| Component | Purpose | Linux | Windows |
|---|---|---|---|
| agent | The agent to run inside aa CI job | agent | agent.exe |
| wsproxy | SSH proxy command that can be directly used by ssh | wsproxy | wsproxy.exe |
| tcptows | TCP to WS tunnel for allowing regular SSH and SFTP clients to connect to converge | tcptows | tcptows.exe |