Hacktivate

I am working on a project here in the Philippines that uses computers to help rural health care workers capture and use data more effectively. I am really trying to figure out how to make the hardware configuration as easy and off the shelf as possible so we can quickly scale this up once we have the kinks worked out. Unfortunately off the shelf appears to mean a Windows XP netbook with Limewire and who knows what else pre-installed. For various reasons some of these virus magnets can't be wiped because they are owned by someone else.

I have written a script that creates an efficient, appliance-like, ubuntu client. It is customized to automatically connect to the right wireless access point, and it is bundled with a firefox profile that starts on boot and which has a full screen plugin pre-installed, the home page pre-configured, and a nice little plugin called 'try again' that automatically refreshes the page if the connection is lost. Of course, this lovely little piece of work is useless if I can't install Ubuntu.

That is until today, when I figured out how to use remastersys to create a custom bootable Ubuntu. Remastersys creates a bootable iso of your currently installed Ubuntu that you can put on a flash disk (with USB Startup Disk Creator) and boot from and use without making any changes to the hard drive. It's like the Ubuntu install disk but with all of my carefully crafted magic. So now I can convert the useless windows bricks into kick ass appliances without ruffling any feathers just by plugging in a USB stick. Removing the disk and rebooting will return them to their original spambot state. And if they ever decide that the Ubuntu setup is superior (and they will of course) the bootable disk has an option to install onto the hard drive. My USB bootable ubuntu appliance is here in case anybody wants it (but you might as well build your own). Oh, one last trick: use VirtualBox to create your ideal Ubuntu. It makes it really easy. VirtualBox snapshots are helpful in crafting the perfect, compact and clean image (but ideally you should use scripts so that you can repeat anything you do later).

A nice day's work if I do say so myself!

I am currently working on a script to automate the process of setting up mysql database replication. I followed various tutorials but I always got stuck here:

mysql> SHOW MASTER STATUS;
Empty set (0.00 sec)

Endless googling was no help (which is why I am blogging this). Eventually I realized that /etc/mysql/my.cnf was context sensitive, meaning that I couldn't just append the replication configuration to the end of the file. This meant I needed to insert the configuration into the appropriate place in the file. This meant inserting multiple lines of text into the middle of the file. Eventually I came up with the following:

(Update) I used to do this with ruby, but I switched to perl since ruby isn't installed by default:



MYSQL_CONF_ADDITIONS="
# ----------------------------------------
# Allow connections from all addresses
bind-address = 0.0.0.0
# ------------------------------
"

perl -i -p -e "print '${MYSQL_CONF_ADDITIONS}',$_='' if \$_ =~ /bind-address.*127.0.0.1/)" /etc/mysql/my.cnf

Hopefully this will be useful to somebody, someday, somewhere.

I highly recommend this excellent article in the NYTimes about how we can use data to create healthcare protocols that dramatically improve outcomes and reduce overall costs. (This is what we were trying to do in Malawi and what I am trying to introduce in the Philippines)

Here are the key points that I want to remember:

To enter mainstream use, any such treatment typically needs to clear a high bar. It will be subject to randomized trials, statistical-significance tests, the peer-review process of academic journals and the scrutiny of government regulators. Yet once a treatment enters the mainstream — once we know whether it works in certain situations — science is largely left behind. The next questions — when to use it and on which patients — become matters of judgment, not measurement. The decision is, once again, left to a doctor’s informed intuition.
...
“Guys, it’s more important that you do it the same way than what you think is the right way.”
...
Whenever possible, the guidelines are also embedded in the hospital’s computer system. Doctors and nurses are presented with a default choice — how much of a given drug to prescribe, for example — and have the option of overriding it. Most important, the electronic records system allows both committees and doctors to track patient outcomes.
...
He could not simply tell Intermountain’s doctors what to do, no matter how much research he brought to bear. Doctors have a degree of professional autonomy that is probably unmatched outside academia. And that is how we like it. We think of our doctors as wise men and women who can combine knowledge and instinct to land on just the right treatment.
...
Perhaps the clearest example is the Pronovost checklist. As many as 28,000 people in this country die each year from infections that come from intravenous lines. Several years ago, Peter Pronovost, a Johns Hopkins physician, developed a simple list of five steps that intensive-care doctors should take before inserting an IV line, in order to prevent the introduction of bacteria. The checklist reduced the infection rate to essentially zero at 108 hospitals in Michigan where it was adopted. Pronovost published the results in The New England Journal of Medicine
...
But in our current health care system, there is no virtuous cycle of innovation, success and expansion. When Intermountain standardized lung care for premature babies, it not only cut the number who went on a ventilator by more than 75 percent; it also reduced costs by hundreds of thousands of dollars a year. Perversely, Intermountain’s revenues were reduced by even more. Altogether, Intermountain lost $329,000. Thanks to the fee-for-service system, the hospital had been making money off substandard care. And by improving care — by reducing the number of babies on ventilators — it lost money. As James tartly said, “We got screwed pretty badly on that.”
...
As long as doctors and hospitals are paid for each extra test and treatment, they will err on the side of more care and not always better care. No doctor or no single hospital can change that. It requires action by the government.
...
Yet somehow, both doctors and patients have come to imagine that a physician can accomplish far more than any human being reasonably can. As a result, modern medicine is accomplishing far less than it reasonably should.

Noise blocking earmuffs...check
Music...check
Children locked away in kiddie jail...check

Using autossh and reverse ssh tunneling to bust out of NAT'd networks and firewalls automatically.

In another lifetime I blogged about how to reverse an ssh connection. This was by far my most popular post ever. It earned enough in google ad money to cover my hosting costs. Those were the days. Those days are gone, and now lots of people have blogged about using reverse ssh tunnels. It's time to take it to the next level and make reverse ssh tunnels easy and ubiquitous.

I want all of my machines to be setup with an ssh tunnel whenever one of them finds an internet connection. That way I can always access all of my machines no matter where I, or they are the world. I want this even if they are in some high security firewalled corporate prison or if they are sharing the same IP address as the other million NAT'd users tethering internet through their phones. This should happen automatically and without fuss.

To do this you need to have ssh access to a machine that can be seen from the internet, and the ssh daemon machine needs to have the GatewayPorts option set to yes (this is not the default). See my old post for more information.

Here's the recipe.

(Update! I have made this much easier by scripting the whole process, see the bottom of the post!)

Create a file:

/etc/network/if-up.d/reverse_ssh_tunnel

(you are using Ubuntu, right?)

Put this in it:

#!/bin/sh
# ------------------------------
# autossh reverse tunnel on boot
# ------------------------------
# See autossh and google for reverse ssh tunnels to see how this works

# When this script runs it will allow you to ssh into this machine even if it is behind a firewall or has a NAT'd IP address.
# From any ssh capable machine you just type ssh -p $PORT_MIDDLEMAN_WILL_LISTEN_ON localusername@middleman

# This is the username on your local server who has public key authentication setup at the middleman
USER_TO_SSH_IN_AS=change_me

# This is the username and hostname/IP address for the middleman (internet accessible server)
MIDDLEMAN_SERVER_AND_USERNAME=luser@example.com

# The following two numbers can be whatever you want, but need to be unique if you have multiple reverse ssh tunnels
# Port that the middleman will listen on (use this value as the -p argument when sshing)
PORT_MIDDLEMAN_WILL_LISTEN_ON=11829

# Connection monitoring port, don't need to know this one
AUTOSSH_PORT=27554

# Ensures that autossh keeps trying to connect
AUTOSSH_GATETIME=0

export AUTOSSH_PORT AUTOSSH_GATETIME

su -c "autossh -f -N -R *:${PORT_MIDDLEMAN_WILL_LISTEN_ON}:localhost:22 ${MIDDLEMAN_SERVER_AND_USERNAME} -oLogLevel=error -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no" $USER_TO_SSH_IN_AS


Make sure that you fill in the values to match your own logins (send me a comment if you are confused about what to put in).

Now whenever the machine acquires an internet connection it will run the above script (that is why we put it in /etc/network/if-up.d). Then from any internet connection you can ssh -p 11829 USER_TO_SSH_IN_AS@MIDDLEMAN and you will get forwarded to your own machine.

You should put this on all of your machines. That is all.

Oh one more thing. For this to work you need to have passwordless public key authentication working between your machine and the middleman. Use this:


ssh-copy-id


It will take care of all of the gory details of copying and concatenating your keys so that you can ssh in without typing anything.

This blog post is a mess and needs some serious revision, but I needed to post it, if for no other reason so that I could find the info when I needed it!

Update

I put everything you need to set this up into a script on github. If you have a server out on the internet with the GatewayPorts option turned on, then all you need to do is the following:


wget http://github.com/mikeymckay/reverse_ssh_tunnel/raw/master/setup_reverse_tunnel.sh
chmod +x ./setup_reverse_tunnel.sh
sudo ./setup_reverse_tunnel.sh
: