Tensorflow Python Virtualenv setup

Learning a new technology can be challenging, and sometimes setup can slow you down.

I wanted to experiment with machine learning and just follow along with a a few tutorials.  But first, I had to get a compatible environment up and running.

So I thought I’d document it here, and after several bookmark/footnote tweets,

### install pyenv and virtualenv 
brew install pyenv
brew install pyenv-virtualenv

### initialize pyenv
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

### install python 3.6 with pyenv
pyenv install 3.6.5

### create a virtualenv for using tensorflow
pyenv virtualenv 3.6.5 tf
pyenv activate tf

### install dependencies in virtualenv using
pip pip install --upgrade pip
pip install tensorflow

### optionally use jupyter web interface
pip install jupyter
jupyter notebook

### use tensorflow in your app
import tensorflow as tf

I created a gist.


Here’s the great intro to Deep Learning video I followed along with:

I’ll follow up with a quick screencast video showing how the steps look together for those who want to see it.


How do you practice TDD for Android development projects?

TDD is more of a strategy then a certain process or tool. The idea is that testing drives your development — writing tests first, making sure tests pass to indicate development is complete, and continuously testing on each change to make sure no regressions or new bugs slip in.

You can use Espresso, Robotium, or UIAutomator directly for automating the mobile app but testing the UI is inherently slow, can be brittle, and may not be possible (or easy) to write UI (or end-to-end) tests while an app is under development. The UI may not be testable, or back end services may not be complete at early stages.

With test driven development, you want to use your tests to inform what you develop. This informs what you should develop first, and it helps you to write your application in a way that is testable.

If you have some feature that needs tested — for example: delivering different size media (images & video) based on available bandwidth and screen size, testing this with the UI seems to make sense, since it is a UI feature.

But try writing your test the way you want it to look, not the way it actually behaves in the app. Start with your assertion:


Now we try to satisfy that

First we need to get our values. Where does deviceScreenWidth come from? How do we determine imageWidth?

imageWidth is probably sent to the response processor so that when it sends the image URL it resizes the image — or selected the appropriately sized image.

That’s a design decision that’s already being influence by our tests. Maybe we want standard sizes — small, medium, large instead of trying to support every possible pixel width. Maybe isEqualTo should test within a range instead of just equal.

For deviceScreenWidth we need some representation of our device that includes it’s screen size. Do we get it from the userAgent or does the device send DisplayMetrics via an API? Is it passed from a service or a lookup table? Maybe we need a test of the function that passes a device identifier from the userAgent and calculates based on known values.

Now we know what code to write — and another test to write.

This can be a bit of a rabbit hole, but we don’t have to tackle everything at once.

In our unit test we just need to have an imageWidth and a deviceScreenWidth. We can make a note of what functions and parameters are needed to get this information, but for now we can just implement the functions immediately needed — and even make our first test pass by having those functions return hard coded values.

A nice simple test might look like this:

public void testImageCalculator() 
    device = new DeviceMetaData(SAMSUNG_GALAXY_S6);
    deviceScreenWidth = device.getDisplayMetrics(device).screen.width;
    imageWidth = getImageSizeForDevice(deviceScreenWidth);
    assertThat(imageWidth).isBetween(deviceScreenWidth, mediumDeviceMaxWidth);

Now we know what we need to develop next — the functions that make this test pass. A DeviceMetaData container class, something that gets display metrics for the device, and what we really care about (at this time) — the getImageSizeForDevice() function.

NOTE: This was originally an answer to a question on Quora



Testing web & mobile app interaction with Selenium & Appium


Sometimes there is a need to test the way two different apps interact or work together.

Say you have a mobile app for a coffee shop. When you place your order with the app, a notice shows up on the order terminal for the barista.  They can then fulfill the order and have it ready for you — no waiting in line.

As a customer
I want to place an order for coffee on my iPhone
So that it's ready when I get to the coffee shop

But if they can’t fulfill the order (maybe they’re out of caramel-soy-macchi-whatever mix) they can let the customer know so that they can cancel or place a new order without waiting to find out that their order is not available.

As a barista
I want to notify customers when their order can't be fulfilled
So that they can change or cancel their order

There are two different apps here, and two different actors.  This can make the test challenging.  The obvious solution is to automate both apps at the same time (the mobile app for the customer and the web-based point of sale terminal for the barista.

Your test might looks something like this:

The problem here is that coordination between the two apps can be tricky.  Synchronization and timing isn’t guaranteed and I’m not sure if the explicit waits will always handle this.

Also, it requires standing up both environments and making sure that the mobile app can communicate with your web app.  It can get tricky.  Not to mention it will be inherently slower and the odds of random failures increases.

Another thing you can do is test the majority of use cases independently.  This is hinted at by our two stories above.  One for the barista (web app) and a separate one for the customer (mobile app.)

Unless you have a really unique architecture, it’s likely that the two apps don’t actually know anything about each other.  They probably communicate through web services with a shared back end database or message queue.

Really, what you want to do is test each app independently and how it interacts with the service.  The service can be mocked or stubbed for some use cases, but for end-to-end tests, it makes sense to use the service.

So your test will now look something like this:

This requires a clear API and may require some backend interaction that is not normally exposed.  But the test is much cleaner (and reliable) and if exposed services require additional security you can have a separate test API endpoint or authorization token that enables the additional functionality.  In this case, that shouldn’t be necessary.

You may still want to perform a few end-to-end sanity tests to make sure the environments are communicating correctly and are compatible, but the number of these tests can be greatly reduced — and the speed and reliability of your test suite improved.

The Practice of Programming – Preface

The Practice of Programming
Brian Kernighan & Rob Pike


Have you ever…

wasted time coding the wrong algorithm
used a too complicated data structure
missed an obvious problem despite testing
spent all day looking for an obvious bug
needed to make a program run 3x faster and use less memory
run a program on different computer than it written on
tried to make a modest change in someone else’s program
rewritten a program because you couldn’t understand

what a great set of questions

these practices are not taught:
design alternatives

underlying principles:

clear description of the applicable target solution

tools and notations that let the computer assist you
automation – let the machine do the work for you
free you from mundane tasks

learn design, debugging, testing, performance improvements independent of language, os, or programming paradigm

Chapter summaries

Chapter 1: Programming Style
well written programs are better than badly written ones
– fewer errors
– easier to debug
– easier to modify
important to think about style from the start
use language appropriate idioms

Chapter 2: Algorithms and Data Structures
review common ones that show up in most programs
more complex algorithms should evolve from building blocks so master the basics

Chapter 3: Implement alg & ds in different languages
compare expressiveness and performance

Chapter 4: Designing interfaces
between users, programs, subsystems
evolution of small library for parsing a data format
small example illustrate concerns of interface design
– abstraction
– information hiding
– resource management
– error handling

Chapter 5: Debugging
debugging is inevitable
systematic & effective debugging strategies & tactics
signatures of common bugs
patterns in debugging output indicate where a problem is

Chapter 6: Testing
attempt to develop a reasonable assurance that a program is working correctly
and that it stays correct as it evolves

Chapter 7: Efficiency & Performance
most programs are “fast enough” on modern hardware
orderly way of making programs use resources efficiently
make sure programs remain correct and sound as perf improves

Chapter 8: Portability
Successful programs will see environment change underneath them
Goal of portability is to reduce maintenance by minimizing change needed to adapt to a new environment

Chapter 9: Language
general purpose vs specialized languages
guide implementations
programs that write programs

Code: C, C++, Java
descendants of C

Your task is to do the best you can with the tool you have

Examples may be in C even though another language might be better suited to a particular problem.

Reviewing classic computing literature

I’m going to be reading through and commenting on some of the classics in the field.  I’d welcome anyone who wants to join along and do a kind of virtual book club where we share our thoughts on blogs, twitter, etc.

I’m starting with “The Practice of Programming” by Brian Kernighan and Rob Pike.  It’s a nice general primer on programming style with concrete examples.   After that I’ll dive into the original K&R “The C Programming Language” working through the exercises and looking for the principles mentioned in the later book.  And then I’ll take a look at “The Unix Programming Environment” to round out the trifecta and consider how it has influenced modern systems.

And then I’ll look for some other classics, and maybe some modern books.  Open to suggestions.

Planning about 1 book every month or 2.  Spending just 30-60 minutes / day 3-4 days a week on each book, not going to deep, but looking for little insights and comparison founding principles with modern views.

Let me know if you’re interested in reading along, maybe a hangout or chat room or discussion board.

Blockchain is essentially a signature

Blockchain is essentially a signature.

You need to think about the value of a signature. What is the trust value of of a pen on paper signature? At the local grocery store, they don’t accept checks anymore. That should tell you that. Although it’s not just the quality of the signature, but the cost to verify, and the quality of the verifying authority in this case.

What about a PGP signature? From 512-4096 bits — pretty much verifiable, as long as you can keep your key secret — but there isn’t a good trusted authority, and no standard acceptable interchange. You can put on your tinfoil hat about this.

How about TLS? HTTPS with 2mB is good enough for most transactions, usually with a tolerance of a few hundred dollars. Browser manufacturers and SSL certificate authorities are good enough here for almost everyone. You can get that for around $50 but some people pay thousands of dollars for greater peace of mind. There’s no accounting for psychology.

So now we have blockchain currencies — which try to solve both the problem of the trust of banks, and the monopolies of browser CAs. But do both poorly. And they don’t solve the fundamental problem — that of a clearinghouse for exchange. You still need a trusted central source of exchange. Mt Gox proves that. And using the blockchain to clear a blockchain is a obvious mobius strip paradox.

What you need to do is figure out the tolerance for risk at certain value. Or stated another way, the cost of “breaking” the security of a certain signature. For personal checks, that used to be around $200, before banks became ineffective at both security and clearing funds.

For cash, the risk value is about $100. If you’re at a gas station that says “we don’t accept $100 bills”, you’ve experienced this.

In truth, a simple public MD5 sum should be approximately equivalent in risk to a paper signature or money. It’s not worth trying to break — or counterfeit for less than about $100-200. But since there’s no medium of exchange, maybe half that value — MD5 should be safe for up to $50.

A SHA1 signature should probably double that. With an additional byte count checksum, it should double again. So call it $200.

PGP should be as good as SSL — sufficient for transactions up to $1000. What’s missing is a CA. PGP with a CA should be good for several times that. Maybe $10K transactions.

Obviously something more is needed for large sum transactions — bank accounts, vehicles, property. But all we have in place currently for these sorts of things are contract paperwork, wire transfers — and here’s the barrier to entry — government enforcement. Try lending money over $100K without being a part of the system and see where you get.

The force of law includes both your lifetime earnings, and your freedom. That’s the collateral. This is what enables transactions over $10K. And you have to have control over the law to enforce that. You have to be a “bank”.

For smaller amounts though, below $100, possession of a cell phone is sufficient. Think venmo, uber. With paypal, they have your identity, so you can get larger amounts, gradually increasing as your trust — and stake in paypal increases. Up to where you need a merchant account — where you are back to providing collateral for the powers that be.

So it comes down to three things:

1 – trust
2 – risk
3 – collateral

Blockchain cryptocurrencies only address #1, and do much more inefficently, and only marginally more securely than a simple cryptographic hash. And the real show stopper, is that there is no means of exhange, to enter or exit the system. It requires total buy in from everyone — in other words, to be declared FIAT, for it to even succeed as a medium of exchange. While still not solving the obstacles of risk or collateral.

Alternatives to Wordpress?

I’d like to find an alternative to WordPress.  But I want a CMS, not a hosted WYSIWYG website builder (like Wix), and not a static site generator (like Ghost).

Here are some of the features of WordPress that I see as most valuable:

  1. Editing pages / posts — for convenient content delivery
  2. Admin interface — for management by non-developers
  3. Themes — for consistent and changeable layout and appearance
  4. Plugins — for added functionality
  5. Analytics — for tracking visitors and conversion
  6. SEO — for improved search engine results

Here are some things that I think WordPress is lacking or weak:

  1. Development — WordPress is spaghetti code and developing themes and modules is clunky
  2. Customization — Because it’s so clunky, it’s difficult to customize layout and functionality.  A WordPress theme / module developer is a special niche.
  3. Backups & Versioning — WordPress still doesn’t have a reasonable solution here.  The root of the problem is that all content is stored as blogs in a (poorly designed) MySQL database.  Keep content out of the DB and put it back in files.
  4. Static — WordPress is always run on PHP with MySQL.  You should be able to export your site, and it should still work (with graceful degradation) if either your database or your CGI module goes down.

I would like a tool that is good for building documentation heavy websites, blogs, business sites, marketing landing pages, and some e-commerce capabilities.  You should be able to connect to APIs, build modules with templates that connect to those APIs and embed components on pages easily.  A plugin architecture for sharing modules / components is also desirable.

The site should consist of static HTML pages that can be generated from templates / markdown / other tools etc.  So having a route / map is important — but the site should be able to function using file / directory structure from a static webserver at it’s most basic.  Some functionality, some data, and some layout and dynamic content may not work, but your site should still be up, you should still be able to navigate and access content even if everything except static web serving is down.

Then there should be a front controller that is injected before each page that can add additional functionality such as analytics, dynamic routing & content, etc.  This controller would also enable not just modules but the admin interface.  Ideally, as much as possible is handled via javascript added to the pages (optionally even to the static pages) to handle things like analytics, dynamic interaction, fetching content / data via services, etc.

Then there should be the tools that manage compiling templates, incorporating modules, etc.  This should be able to be done offline, or on demand while the server is running — for instance while editing a page / layout / content / plugin via the admin UI.

Is this too much to ask?