Docker Bash shell and Ctrl+P

If you are running an interactive Bash shell in a Docker container running Linux, the Ctrl+P key does not perform the previous-history command, which fetches the previous entry in the command history entry and repeats it at the current prompt. But if you press Ctrl+P twice, it happens to work.

The reason for this is that Docker's interactive mode assigns "Ctrl+P Ctrl+Q" to detach from the interactive Docker session. To fix this, I customize the detach keys using --detach-keys command line option:

$ docker run -it --detach-keys 'ctrl-z,ctrl-q' ubuntu bash

Ctrl+P now works as expected!

Async Job Lessons

On my project at work, we are improving our HTTP request processing times by extracting expensive update operations and offloading them to a distributed job queuing system that can execute them asynchronously. While this has helped reduce our system's response times, we have run into a couple of issues that are a fallout of the asynchronous design.

1. Consider an async job that is triggered while a database transaction is in progress. The transaction inserts row A. The async job will read row A as input, perform a calculation on it, and then record its result elsewhere. If the async job runs before the transaction is committed, the job will fail, since row A is not yet visible outside the transaction. One solution is to delay enqueuing the job until after the transaction is committed. However, this may be difficult to implement. The advantage of this approach is that it can decide never to invoke the async job ever if the transaction aborts. A less desirable solution is to enqueue the job immediately, but to delay the execution of the async job a fixed amount of time (a feature commonly supported by job queuing systems), but then one must decide what a reasonable delay should be. A third solution is to simply rely upon the job being retried, assuming the job queuing system supports retries. This has the possible downside of errors being reported when the job initially runs, which may be unnecessarily alarming depending upon the configuration of the ops environment.
2. Async jobs that perform updates can suffer from race conditions. If multiple jobs of the same type are invoked in quick succession, and the ordering of async jobs is not guaranteed, then incorrect updates may be made. In particular, the ordering might be reversed if the first job suffers a transient failure (e.g. network communication error) and gets retried after the second job runs successfully. If the two jobs are recording different values, the recorded value will be incorrect after the retried job succeeds, since the value it records is a stale value. One solution is to have such jobs calculate the correct value at the time the job run. If implemented this way, the ordering will not matter, since the last job to run will always calculate the correct value using input values that up to date. As a corollary, it is a bad design to have a job record a value that is provided to it via a parameter at the time of invocation, since that value may become stale.

RSpec 3.x Goodness

RSpec 3.3 was released "way back" in June, but I just looked into it, and it provides a new "aggregate_failures" feature. This allows RSpec to run test setup code once for multiple expectations. This moves away from the "one expectation per example" pattern while still running all tests (normally, if you include multiple expectations in a single test, the first failing expectation will short circuit the rest). Used properly, I would expect massive performance increases with feature specs, in particular! In a similar vein, the 3.x series also introduces "compound expectations" for checking a result value against multiple criteria all at once via "and" and "or" composing methods. And if you haven't seen 3.0's "composable matchers", have a look!

Kill Process Group

When a UNIX process spawns child processes, they all belong to the same process group. And you can kill the whole process group at once using kill -9 -. The negation symbol in front of the PGID value is the key here. How did I not know this before? I always grep'd the output of ps and killed the individual processes.

Long vs Short Command Line Options

It's simple really. Don't use short options when invoking commands from scripts. Thanks.

Code Review

After watching RailsConf 2015 - Implementing a Strong Code-Review Culture, I realize my own code review process is perhaps much more heavy weight than it could be. In particular, I realize that I generally perform a full QA check on each pull request, doing my own manual testing, etc. I also focus heavily on finding bugs, unlike the presenter of this talk. Of course, if I scale back my CR efforts, I know there will be lots of defects that will make it into production, as my current team does not have a QA team at its disposal.

Software Stymied by a Single Schema?

Most commonly, a software application's persistence layer is a relational database. And so the application's software architecture becomes intricately tied to a single, underlying physical relational schema. Each table is represented by a domain model class, which is tied to a physical table via an ORM framework (Hibernate, Rails ActiveRecord, etc.) that together comprise the domain model.

Over time, the domain model and schema evolve and grow to accommodate additional features (business requirements). In turn, tables and their associated domain model classes take on additional attributes/columns and associations/foreign keys to support the persistence of data needed by the new features. Often, a non-trivial feature will require updates and additions to the domain model and schema that span numerous classes and tables. Before long, the features that comprise the application have code that is spread across the domain model. And conversely, a given domain model class will include attributes and code to support numerous and often unrelated features and business requirements.

The problem is that the application's class structure and physical schema can end up bearing little resemblance to the feature set and business requirements of the application. The mapping between the business requirements (features) and the class design of the application becomes a many-to-many relationship.

One undesirable outcome of this is that multiple features may end up depending upon many of the same classes and attributes in the domain model. And thus changing the usage, semantics, or implementation of any given model or attribute for one feature involves understanding its usage and impact of any other feature that depends upon it as well. Conversely, studying the applications domain model and physical schema does not directly reveal the underlying set of features and business requirements the comprise the application.

Is there a better way to structure our applications to maintain a more direct mapping between the implementation and persistence schema of the feature set and business requirements?

Perhaps an application should be written as a set of mini-applications, where each of these smaller implementations directly implements a single feature or business requirement.
Those paying attention to recent developments in software architecture trends might cry out "use micro-services!" And indeed, the single-responsibility tenet of this architectural pattern is in fact what I am describing here. But note that I am not concerned specifically with the distributed deployment aspect of this pattern, since my concerns apply to distributed and "monolithic" deployments equally. Regardless of how the application's code is structured and deployed, inevitably the disparate feature implementations require access to shared data. For example, the identity of a "user" must be consistently represented across these multiple feature set implementations. Even if we find an appropriate way to structure the highest-level layers of an application to have a clean, one-to-one mapping with the application's feature set, we still end up having a single persistence layer that becomes a catch-all repository for the full set of features. In other words, the schema becomes the union of mini-schemas that might otherwise be needed by each individual feature set implementation (or "micro-service", if you prefer).

And so we arrive back at the original problem posed herein. Namely, how do we maintain a persistence structure that cleanly maps to the individual feature sets and business requirements of the application?

Is there a way to maintain individual schemas--one per feature--where the previously shared data is instead redundantly stored and structured to singularly support the needs of one and only one feature? This flies in the face of normalized database design tenets. Clearly, without significant additional work, our mini-applications' persistence stores will grow out of sync. Both the schemas and the data contained will end up as very different representations of core domain concepts and domain instances. All the benefits of normalized database design are lost.

But might we be able to free ourselves from the strict rules of normalized database design? Can we develop a synchronization layer to guarantee that necessary and specific constraints are satisfied between the disparate data stores? Can we specify these constraints in a way that guarantees the data can still be used in future, unknown capacities? This after all, is perhaps the greatest promise of the relational model. But can we confidently move past this "plan for the future" design mentality? And if we do, will our applications' architectures benefit from these simpler partitions of both logic and data structure?

I hope to continue my research and thoughts on this matter, since I believe it as the core of the software complexity problems the plague classic application architectures today.

Postgresql: Database Quick Copy

Postgres' database creation commands allow a new database to be cloned from an existing "template" database, including both the schema and the data. As there is nothing special about a "template" database, you can use any existing database within the same database cluster as the source database. This can be much faster than a dump and load operation. This is done simply by using one of the following commands (copied from the Postgres documentation for your convenience):

To create a database by copying template0, use:
CREATE DATABASE dbname TEMPLATE template0;
from the SQL environment, or:
createdb -T template0 dbname
from the shell.

Ignoring method invocations with RSpec message expectations

I often find myself with RSpec test examples that are verifying that log output meets expectations, but where the test is only concerned with verifying a single, specific log message has been generated. But usually the logger is receiving additional output that the test is not concerned about. Unfortunately, this output must still be "expected", lest the test will fail. There is (apparently) no way in RSpec to write an message expectation with specific arguments that should occur "at some point", when the message is also received at other times with different arguments.  For example this simple test example fails if logger.info is called more than once by the code being tested:

it "logs a specific message" do
  logger.should_receive(:info).with(/text that must be logged/)
  invoke_your_test_code()
end

RSpec will complain with something like:

expected: (/text that must be logged/)
     got: ("starting process...")

Writing and maintaining the code that explicitly expects all of the other logging calls is cumbersome to write and maintain, and makes the test fragile to log output changes that are not of concern to the test.

However, I found that one can gracefully ignore this extraneous log output by using the any_number_of_times expectation method in conjunction with the specific expectation to ignore all of the other method invocations, as follows:

it "logs a specific message message" do
  logger.should_receive(:info).with(/text that must be logged/)
  logger.should_receive(:info).any_number_of_times
  invoke_your_test_code()
end

The any_number_of_times expectation must occur after the specific expectation that is being verified; otherwise the expected log output will be "consumed" by the any_number_of_times expectation, and the test will fail.

Two Things Your Configuration Management Should Make Easy

For a given story/issue/ticket, can you produce the associated code diff?

On your project, can you determine the story/issue/ticket associated with the last change that was made for a given line of code?

If your configuration management system makes it possible to do these things, can you do it quickly? Or is it merely "theoretically possible"?

[http://www.sqlite.org/talks/wroclaw-20090310.pdf, slide 141 "Situational Awareness in CM"]

Notes: "The Trouble With Types", Martin Odersky Presentation

http://www.infoq.com/presentations/data-types-issues

Good designs are:

• discovered, not invented.
• opposite of random

Strong typed languages help produce good designs. (I couldn't agree more, after spending almost 2 years now in Ruby space.)

Patterns (abstractions) & Constraints (types)

Type systems should be 1) precise, 2) sound, 3) simple

Odersky addresses issue of Scala's type system complexity, admitting that the complexity comes out of the large combinations of typing features that arise from the set of possible combinations of its modular (OO) and functional typing features.

DOT (Dependent Object Type calculus) & Dotty (experimental language): Simplify typing complexity by focusing on just supporting the module typing features internally, i.e., in the compiler.

It's comforting to know that the parts of Scala I've found most confusing (e.g. when do I define a class using 'class MyClass[T]' or 'class MyClass { type T }') are at the core of what Odersky is trying to simplify.

A Few Noteworthy UNIX Commands

A few noteworthy UNIX commands I've used recently:

• parallel: Utilize your multiple {core,cpu}s!
• fdupes: "Finds duplicate files in a given set of directories"
• s3cmd sync: Like rsync, but with S3 as the file/dir destination

As it turns out you don't need much more than this to backup your photo archives to AWS S3.

I used parallel to speed up various batches of S3 operations whose per-request latency is noticeable for large batch sizes.

Note that s3cmd sync will not copy symbolic links to S3, but it will copy hard links, so fdupes -H can be used to eliminate uploading of these redundant files.

Database Optimization: Query Performance vs. Request Performance

When it comes time to optimize your web application's database performance, there are (at least) two types of tools you can leverage. You can use an analysis tool like pgFouine, which is focused on individual query performance, or a tool like Scout or New Relic, which is focused on request performance.

pgFouine is fantastic at isolating any single query that, in aggregate, is slowing down your application the most. This is a function of the query's invocation count multiplied by its average execution time. Of course, it will report not just the single most expensive query, but the top N most expensive queries. (The tool will also will show you the most frequently executed queries and the overall single slowest queries, but these metrics are less useful for determining what queries to focus on).

But what if the top N queries are all taking roughly the same amount of time? Which one(s) should you try to optimize or eliminate? pgFouine cannot help you make this decision intelligently. Instead you need a tool that will allow you to focus on performance at the level web requests.  Consider that each request your web application handles is likely comprised of many queries. So if you have one type of request that is taking 80% of your web application's processing time, then you probably want to see the full set of queries that together are causing these requests to be non-performant. Perhaps you can eliminate many of these queries, with an improved application design. This is when a tool such as New Relic (or Scout's "Application" feature) becomes invaluable.

For each type of request, you can see the most expensive queries, but more importantly the full set of queries that are being issued for a given request. (These tools are equally useful for finding application layer code that is non-performant, but I'm only concerned with database performance here.) With a request-level view you can start to evaluate your overall design to determine whether you can eliminate certain queries altogether. For example, you might realize that you are issuing two similar but different queries that can be combined into a single query. pgFouine might show that these two queries are equally performant, while a request-level analysis tool will show you that they are being executed side-by-side while serving a single request. This is the hint one needs to start understanding where optimizations can be made at a design level higher than an individual query.

PostgreSQL Column-to-Row Transposition

I recently had a need to generate a geolocation history of user activity. The result set needed to be a linear history of users' activities, with each row consisting of a user identifier, activity type, location, and timestamp. Unfortunately our database schema stored the location and timestamp of three different types of activities across 3 separate pairs of columns in the same table. To accomplish the required output, I needed to transpose the columns into rows. To accomplish this, I was able to make use of PostgreSQL's array constructor syntax and unnest array function.

From a table with the following columns:

• user_id
• activity1_location
• activity1_timestamp
• activity2_location
• activity3_timestamp
• activity3_location
• activity3_timestamp

I issued the following query:

SELECT user_id, 
unnest(ARRAY['activity1', 
             'activity2',
             'activity3']) as "activity type",

unnest(ARRAY[activity1_location, 
             activity2_location, 
             activity3_location]) as "location",

unnest(ARRAY[activity1_timestamp, 
             activity2_timestamp, 
             activity3_timestamp]) as "timestamp";

The unnest function generates multiple rows, one row per element of the specified array. This produces a result such as:

user_id | activity_type | location    | timestamp

--------+---------------+-------------+--------------------

      1 | activity1     | address1    | 2012-03-13 00:00:00

      1 | activity2     | address2    | 2012-03-13 00:00:01

      1 | activity3     | address3    | 2012-03-13 00:00:02

      1 | activity1     | address4    | 2012-03-13 00:01:00

      1 | activity2     | address5    | 2012-03-13 00:01:01

      1 | activity3     | address6    | 2012-03-13 00:01:02

      2 | activity1     | address7    | 2012-03-14 00:00:00

      2 | activity2     | address8    | 2012-03-14 00:00:01

      2 | activity3     | address9    | 2012-03-14 00:00:02

...

I learned about these PostgrSQL array functions from this highly recommended slide presentation, "Postgres: The Bits You Haven't Found".

Quicker Code Reviews

This year I resolve to spend less time doing code reviews by only reviewing the test code. This should work, right?

Android "External Storage" Poorly Named

As a user of an Android device, I've always been a little confused about the various types of data storage that are available, and where exactly my apps are storing their (well really, my) data. There's "internal storage", which I've always assumed is akin to the internal hard drive on a normal computer: durable, persistent storage. And then there's "external storage", which I've assumed meant a removable SD card. So I always assumed that if I were to take the memory card out of my phone and toss the phone in the water, I would at least maintain in my possession all of the data that I've explicitly moved to (or configured apps to automatically save to) "external storage". Well, it turns out that's not quite the case. What Android docs call "external" storage is really just a "non-private" data space. And a removable SD card may or may not be where this non-private storage resides, as it can actually be a partition of the internal storage! Come on Google! Why call this "external storage" at all? It's "non-private" or "public" storage. Please rename it.

Here are the official docs:

Using the External Storage
Every Android-compatible device supports a shared "external storage" that you can use to save files. This can be a removable storage media (such as an SD card) or an internal (non-removable) storage. Files saved to the external storage are world-readable and can be modified by the user when they enable USB mass storage to transfer files on a computer.

It's possible that a device using a partition of the internal storage for the external storage may also offer an SD card slot. In this case, the SD card is not part of the external storage and your app cannot access it (the extra storage is intended only for user-provided media that the system scans).

Subversion Directory Tree Conflicts

Came across this animation on a blog while looking for some answers on how to properly resolve a Subversion tree conflict on a directory. This about describes how I feel at the moment, after having already spent a large part of the day working with merging source code branches. In fact, I often feel like this, when I can't find proper documentation for the software I'm using.

For what it's worth, the "answer" I was looking for was found in the last paragraph here, which tells me that Subversion will be of no help in resolving my particular problem. Joy!

Other tree conflicts

There are other cases which are labelled as tree conflicts simply because the conflict involves a folder rather than a file. For example if you add a folder with the same name to both trunk and branch and then try to merge you will get a tree conflict. If you want to keep the folder from the merge target, just mark the conflict as resolved. If you want to use the one in the merge source then you need to SVN delete the one in the target first and run the merge again. If you need anything more complicated then you have to resolve manually.             

Scala "for" iteration with indexes

In Scala, to iterate through a collection of items while keeping an index, Seq.zipWithIndex:


for (e <- items.zipWithIndex) {
  println(e._1 + " at index " + e._2)
}


(I find this especially useful when writing Scala code that calls into Java library setter methods that are index-based.)

Getting Started is the Hardest Part

Too often, when I'm trying to get started on a small, personal software project, I'm stymied by the time it takes to get the development environment and project infrastructure setup. With a full-time job as a developer, an addiction to cycling, and the responsibilities associated with being the parent of a two-year child, it's hard to find the mental energy and time to work on even a small software idea.  So when I do have an hour of mental energy available, the last thing I want to spend it on is project setup and configuration task. Maven archetypes to the rescue! Archetypes allow you to setup your project nearly instantly, and if you have appropriate Maven support in your IDE, you'll be ready to code within second (okay, minutes). If--and this is a big if--you can find an appropriately up-to-date archetype that provides the exact stack of technologies upon which your project will rely. So far, I don't seem to have such luck (can any one tell me where I can find well designed sampling of Scala-based Maven archetypes?) So instead of trying to start off with someone else's half-baked archetype each time I need to start a project, I've decided to take the time create my own archetype(s) that I can reuse and evolve for my own needs. The following Maven reference page was all I needed to figure out how to generated my own custom archetypes: http://maven.apache.org/archetype/maven-archetype-plugin/advanced-usage.html.

Find Most Recently Modified File

To find the most recently modified file in the current directory tree:

find . -type f -printf '%T@\t%t\t%p\n' | sort -nr | head -n 1 | cut -f 2,3