home

Wash helps you manage your remote infrastructure using well-established UNIX-y patterns and tools to free you from having to remember multiple ways of doing the same thing.

Introduction

Have you ever had to:

If so, then some parts of the following tables might look familiar to you. If not, then here’s how AWS/Docker/Kubernetes/GCP recommends that you do some of these tasks.

List all
AWS EC2 instancesaws ec2 describe-instances --profile foo --query 'Reservations[].Instances[].InstanceId' --output text
Docker containersdocker ps --all
Kubernetes podskubectl get pods --all-namespaces
GCP Compute instancesgcloud compute instances list
Read
Console output of an EC2 instanceaws ec2 get-console-output --profile foo --instance-id bar
Console output of a Google compute instancegcloud compute instances get-serial-port-output foo
An S3 object’s contentaws s3api get-object content.txt --profile foo --bucket bar --key baz && cat content.txt && rm content.txt
A GCP Storage object’s contentgsutil cat gs://foo/bar
Exec uname on
An EC2 instancessh -i /path/my-key-pair.pem ec2-user@195.70.57.35 uname
An a Docker containerdocker exec foo uname
Exec on a Kubernetes podkubectl exec foo uname
On a Google Compute instancegcloud compute ssh foo --command uname
Find by ‘owner’ tag/label
EC2 instancesaws ec2 describe-instances --profile foo --query 'Reservations[].Instances[].InstanceId' --filters Name=tag-key,Values=owner --output text
Docker containersdocker ps --filter “label=owner”
Kubernetes podskubectl get pods --all-namespaces --selector=owner
Google Compute instancegcloud compute instances list --filter=”labels.owner:*”

That’s a lot of commands you have to use, applications you need to install, and DSLs you have to learn just to do some very fundamental and basic tasks. Now take a look at how you’d perform those same tasks with Wash:

List all
AWS EC2 instancesfind aws/foo -k '*ec2*instance'
Docker containersfind docker -k '*container'
Kubernetes podsfind kubernetes -k '*pod'
GCP Compute instancesfind gcp -k '*compute*instance'
Read
Console output of an EC2 instancecat aws/foo/resources/ec2/instances/bar/console.out
Console output of a Google compute instancecat gcp/<project>/compute/foo/console.out
An S3 object’s contentcat aws/foo/resources/s3/bar/baz
A GCP Storage object’s contentcat gcp/<project>/storage/foo/bar
Exec uname on
An EC2 instancewexec aws/foo/resources/ec2/instances/bar uname
An a Docker containerwexec docker/containers/foo uname
Exec on a Kubernetes podwexec kubernetes/<context>/<namespace>/pods/foo uname
On a Google Compute instancewexec gcp/<project>/compute/foo uname
Find by ‘owner’ tag/label
EC2 instancesfind aws/foo -k '*ec2*instance' -meta '.tags[?].key' owner
Docker containersfind docker -k '*container' -meta '.labels.owner' -exists
Kubernetes podsfind kubernetes -k '*pod' -meta '.metadata.labels.owner' -exists
Google Compute instancefind gcp -k '*compute*instance' -meta '.labels.owner' -exists

From the table, we see that using Wash means:

And this is only scratching the surface of Wash’s capabilities. Checkout the screencast below to see some more (and to see Wash in action):

Getting started

Wash is distributed as a single binary, and the only prerequisite is libfuse. Thus, getting going is pretty simple:

  1. Download the Wash binary for your platform
    • or install with brew install puppetlabs/puppet/wash
  2. Install libfuse, if you haven’t already
    • E.g. on MacOS using homebrew: brew cask install osxfuse
    • E.g. on CentOS: yum install fuse fuse-libs
    • E.g. on Ubuntu: apt-get install fuse
  3. Start Wash
    • ./wash

Wash uses your system shell to provide the shell environment. It determines this using the SHELL environment variable or falls back to /bin/sh. See wash shell on customizing your shell environment.

At this point, if you haven’t already, you should start some resources that Wash can actually introspect. Otherwise, as Han Solo would say, “this is going to be a real short trip”. So fire up some Docker containers, create some EC2 instances, toss some files into S3, launch a Kubernetes pod, etc. We’ve also provided a guided tour that includes a Docker application.

NOTE: Wash collects anonymous data about how you use it. See the analytics docs for more details.

Release announcements

You can watch for new releases of Wash on Slack #announcements, the puppet-announce mailing list, or by subscribing to new releases on GitHub.

Known issues

On macOS

If using iTerm2, we recommend installing iTerm2’s shell integration to avoid issue#84.

If the wash daemon exits with a exit status of 255, that typically means that wash couldn’t load the FUSE extensions. MacOS only allows for a certain (small) number of virtual devices on the system, and if all available slots are taken up by other programs then we won’t be able to run. You can view loaded extensions with kextstat. More information in this github issue for FUSE for macOS.

Wash by example

To give you a sense of how wash works, we’ve created a multi-node Docker application based on the Docker Compose tutorial. To start it run

1
2
svn export https://github.com/puppetlabs/wash.git/trunk/examples/swarm
docker-compose -f swarm/docker-compose.yml up -d

If you don’t have svn installed, you can git clone https://github.com/puppetlabs/wash.git and prefix swarm/docker-compose.yml with wash/examples.

This starts a small Flask webapp that keeps a count of how often it’s been accessed in a Redis instance that maintains state in a Docker volume.

When done, run docker-compose -f swarm/docker-compose.yml down to stop the example application.

Navigate the filesystem to view running containers

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
$ wash
Welcome to Wash!
  Wash includes several built-in commands: wexec, find, list, meta, tail.
  See commands run with wash via 'whistory', and logs with 'whistory <id>'.
Try 'help'
wash . ❯ cd docker/containers
wash docker/containers ❯ list
NAME             MODIFIED              ACTIONS
./               <unknown>             list
swarm_redis_1/   03 Jul 19 07:57 PDT   list, exec
swarm_web_1/     03 Jul 19 07:57 PDT   list, exec
wash docker/containers ❯ list swarm_web_1
NAME            MODIFIED              ACTIONS
./              03 Jul 19 07:57 PDT   list, exec
fs/             <unknown>             list
log             <unknown>             read, stream
metadata.json   <unknown>             read

Those containers are displayed as a directory, and provide access to their logs and metadata as files. Recent output from both can be accessed with common tools.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
wash docker/containers ❯ tail */log
==> swarm_web_1/log <==
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
...

==> swarm_redis_1/log <==
1:C 21 Mar 2019 00:02:33.112 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 21 Mar 2019 00:02:33.112 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 21 Mar 2019 00:02:33.112 # Configuration loaded
1:M 21 Mar 2019 00:02:33.113 * Running mode=standalone, port=6379.
...

Notice that tab-completion makes it easy to find the containers you want to explore.

The list earlier also noted that the container “directories” support the metadata action. We can get structured metadata in ether YAML or JSON with wash meta

1
2
3
4
5
6
wash docker/containers ❯ meta swarm_web_1 -o yaml
AppArmorProfile: ""
Args:
- app.py
Config:
...

We can interrogate the container more closely with wexec

1
2
wash docker/containers ❯ wexec swarm_web_1 whoami
root

Try exploring docker/volumes to interact with the volume created for Redis.

Finding with metadata

Wash includes a powerful find command that can match based on an entry’s metadata. For example, if we wanted to see what containers were created recently, we would look at the .Created entry for Docker containers and the .metadata.creationTimestamp for Kubernetes pods. We can find all instances created in the last 24 hours with

1
find -meta .Created -24h -o -meta .metadata.creationTimestamp -24h

That says: list entries with the Created metadata - interpreted as a date-time - that falls within the last 24 hours, or that have the metadata: creationTimestamp in the last 24 hours. See help find for more on using find.

If you want to try this out, or explore more Kubernetes functionality, you can create a Redis server with a persistent volume using Kubernetes in Docker and the following config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
cat <<EOF | kubectl create -f -
kind: PersistentVolume
apiVersion: v1
metadata:
  name: redis
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 100Mi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: redis
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Mi
---
kind: Pod
apiVersion: v1
metadata:
  name: redis
spec:
  containers:
  - name: redis
    image: redis
    args: ["--appendonly", "yes"]
    volumeMounts:
    - name: redis
      mountPath: /data
  volumes:
  - name: redis
    persistentVolumeClaim:
      claimName: redis
EOF

NOTE: find \( -k 'docker/*container' -o -k 'kubernetes/*pod' \) -crtime -1d is an easier (and more expressive) way to solve this problem. This example’s only here to introduce the meta primary.

Listing AWS resources

Wash also includes support for AWS. If you have your own and you’ve configured the AWS CLI on your workstation, you’ll be able to use Wash to explore EC2 instances and S3 buckets.

As an example, you might want to periodically check how many execable instances are running in the AWS plugin (and display that via BitBar):

1
2
3
running=`find aws -action exec -meta .State.Name running 2>/dev/null | wc -l | xargs`
total=`find aws -action exec -meta .State.Name -exists 2>/dev/null | wc -l | xargs`
echo EC2 $running / $total

Or count the number of S3 buckets that have been created:

1
2
buckets=`find aws -k '*s3*bucket' 2>/dev/null | wc -l | xargs`
echo S3 $buckets

Record of activity

All operations have their activity recorded to journals. You can see a record of activity with whistory, and look at logs of individual entries with whistory <id>.

Journals are stored in wash/activity under your user cache directory, identified by process ID and executable name. The user cache directory is $XDG_CACHE_HOME or $HOME/.cache on Unix systems, $HOME/Library/Caches on macOS, and %LocalAppData% on Windows.

Current features

Wash does a lot already, with more to come:

We’ve implemented a number of handy Wash commands (docs):

Core plugins (and we’re adding more all the time, see our docs for how to help):

External plugins:

Several external plugins have already been created:

If you’ve created an external plugin, please put up a pull request to add it to this list!

For more information about future direction, see our Roadmap!

Contributing

There are tons of ways to get involved with Wash, whether or not you’re a programmer!

Come check us out on github, and in particular check out the contribution guidelines and the code of conduct.