+ Converge is a utility for troubleshooting builds on continuous integration servers. + It solves a common problem where the cause of job failure is difficult to determine. + This is complicated further by the fact that build jobs are usually run on a build + farm where there is no access to the build agents or in more modern envrionments when + jobs are run in ephemeral containers. +
+ ++ With Converge it is possible to get remote shell access to such jobs. This works + by configuring the build job to connect to a Converge server using an agent program. + The agent program can be downloaded from within the CI job using curl or wget. + Next, an end-user can connect to the Converge server, a rendez-vous server, that connects + the client and server together. +
+ ++ The setup is such that the connection from client (end-user) to server (agent on CI job) + is end-to-end encrypted. The Converge server itself is no more than a bitpipe which pumps + data between client and agent. +
+ ++ Both ssh and sftp are supported. Multiple shells are also allowed. +
+ ++ There is a timeout mechanism in the agent such that jobs do not hang indefinitely + waiting for a connection. This mechanism is useful to make sure build agents do not keep + build agents occupied for a long time. By default, the agent exits with status 0 when + the first client exits after logging in. This behavior as well as general expiry can be + controlled from within a shell session by touching a .hold file. After logging in, the + user can control expiry of the session as instructed by messages in the ssh session. + When the timeout of a session is near the user is informed about this with messages + in the shell. +
+ ++ In a continous integration job, download the agent, chmod it and run it. +
+{`
+ # linux
+ `}curl http{secure}://{host}/docs/agent > agent{`
+ chmod 755 agent
+ `}./agent --id ID ws{secure}://{host}{`
+
+ # windows
+ `}curl http{secure}://{host}/docs/agent.exe > agent.exe{`
+ `}agent --id ID ws{secure}://{host}{`
+ `}
+ + Above, ID is a unique id for the job, the so-called rendez-cous ID. This should not conflict with IDs + used by other agents. The ID is used for a rendez-vous between the end-user on a local system and + the continuous integration job running on a build agent. If you don't specify an id, a random + id will be generated. + + The agent to the converge server and tells it the ID. Clients can now connect to the Converge + server to establish a connection to the CI job through converge by also specifying the same + ID. + + Communication between + end-user and agent is encrypted using SSH and the rendez-vous server is unable to + read the contents. The rendez-vous server is nothing more then a glorified bit pipe, + simply transferring data between end-user SSH client and the agent which runs an + embedded SSH server. +
+ ++ NOTE: When running the agent on windows, an exit of the remote session using + exit in powershell or command prompt does not terminate the shell completely. + To terminate the client ssh session must be killed by closing the terminal. + Cleanup of remote processes on the agent appears to work properly despite this + problem. It is just that exit of the windows shell (powershell or command prompt) + is not detected properly. +
+ ++ The agent has more options, download the agent and run it without arguments to + see all options. +
+ +wsproxy is a command that can be used as a proxy command for SSH which performs the connection to the
+ remote server. This command needs to be downloaded only once (see downloads below). It does not depend on
+ the converge implementation but only on the websocket standards. Other tools that
+ provide a mapping of stdio to a websocket can also be used instead of wsproxy.
+
+ Next step is to run a local SSH or SFTP client: +
+ +
+ {`
+ `}ssh -oServerAliveInterval=10 -oProxyCommand="wsproxy ws{secure}://{host}/client/ID" { username }{"@localhost"} {`
+ `}sftp -oServerAliveInterval=10 -oProxyCommand="wsproxy ws{secure}://{host}/client/ID" { username }{"@localhost"} {`
+ `}
+
+ + This option is less convenient than the proxy command because it requires two separate + commands to execute. +
+ ++ Local clients can connect using regular ssh and sftp commands through a tunnel that + translates a local TCP port to a websocket connection in converge. See + the downloads section. + This runs a local client that allows SSH to port 10000 and connects to converge using + a websocket connection. +
+ + ++ Next step is to run a local SSH of SFTP client: +
+ + {`
+ `}ssh -oServerAliveInterval=10 -p 10000 { username }{"@localhost"} {`
+ `}sftp -oServerAliveInterval=10 -oPort=10000 { username }{"@localhost"} {`
+ `}
+
+
+ The { username } user above the Converge server and
+ communicated to the agent when the agent is started. This is the
+ username that must be used when setting up an ssh connection.
+ Another way to authenticate is through an .authorized_keys file in the
+ same directory as where the agent is started.
+
+ This can be setup as follows before starting the agent:
+
{`
+ `}# linux {`
+ `}echo "ssh-rsa dkddkdkkk a@b.c" > .authorized_keys {`
+ `}echo "ssh-rsa adfadjfdf d@e.f" >> .authorized_keys {`
+ `} {`
+ `}# windows {`
+ `}echo ssh-rsa dkddkdkkk a@b.c > .authorized_keys {`
+ `}echo ssh-rsa adfadjfdf d@e.f >> .authorized_keys
+
+ + Note that on windows you should not used quotes. +
+ +| Component | +Purpose | +Linux | +Windows | +
|---|---|---|---|
| agent | +The agent to run inside aa CI job | +agent | +agent.exe | +
| wsproxy | +SSH proxy command that can be directly used by ssh | +wsproxy | +wsproxy.exe | +
| tcptows | +TCP to WS tunnel for allowing regular + SSH and SFTP clients to connect to converge | +tcptows | +tcptows.exe | +
- ssh -oServerAliveInterval=10 -oProxyCommand="wsproxy ws{{.secure}}://{{.host}}/client/ID" {{ .username }}@localhost
- sftp -oServerAliveInterval=10 -oProxyCommand="wsproxy ws{{.secure}}://{{.host}}/client/ID" {{ .username }}@localhost
+ ssh -oServerAliveInterval=10 -oProxyCommand="wsproxy ws{{.secure}}://{{.host}}/client/ID" {{ .username }}@@localhost
+ sftp -oServerAliveInterval=10 -oProxyCommand="wsproxy ws{{.secure}}://{{.host}}/client/ID" {{ .username }}@@localhost
- ssh -oServerAliveInterval=10 -p 10000 {{ .username }}@localhost
- sftp -oServerAliveInterval=10 -oPort=10000 {{ .username }}@localhost
+ ssh -oServerAliveInterval=10 -p 10000 {{ .username }}@@localhost
+ sftp -oServerAliveInterval=10 -oPort=10000 {{ .username }}@@localhost