Find Most Recently Modified File

To find the most recently modified file in the current directory tree:

find . -type f -printf '%T@\t%t\t%p\n' | sort -nr | head -n 1 | cut -f 2,3

How to nest single quotes in BASH command

You can't nest single quotes in a bash command, since there's no way to escape it, but you can do something like this instead:

alias a='perl -e '\''print "a\n"'\'

UNIX command to analyze Amazon S3 logs

I wanted to monitor the download activity on a particular file that I made publicly available on my Amazon S3 account.  Here's how I do it with from a bash command-line:

~/dev/s3-curl/s3curl.pl --id= -- https://s3.amazonaws.com/ 2> /dev/null | xpath -e '//Contents/Key/text()' 2> /dev/null | grep '^logs/' | xargs -i ~/dev/s3-curl/s3curl.pl --id= -- https://s3.amazonaws.com//{} 2> /dev/null | grep 'GET\.OBJECT.*'

where and are variables representing the name of your S3 bucket and the filename you'd like to monitor and is the name of the profile you configured in your ~/.s3curl file.

The above command assumes these S3 Logging options for the bucket:

• Enabled is checked
• Target Bucket is the same as the bucket containing the file being monitored
• Target Prefix is "logs/"

In a nutshell, this works by first downloading the file listing of the bucket, then extracting the log file names, then downloading each log file in turn and finally grep'ing them for the download activity (GET.OBJECT).

You'll need to have these command-line tools available:

• s3curl
• xpath

Internal vs. External Events in Databases

A database generally comprises data that records the (instantaneous) state of entities in the real world. If the historical states of the data are of interest, then a history of values may be recorded for these entities.  This history of values will usually be timestamped, and thus this historical list of data values can be considered as recording the occurrence of an event in the real world.  In other words, the real-world event changed the state of a real-world entity, whose updated state must then be recorded in the database.  Let's call these real-world events external events.

But now we must also consider that the process of updating the database to record the new real-world value is an event unto itself. Let's call this an internal event, since it is an event that is intimately tied to the database itself.  This event may be a human operator who manually types in a new value or hardware/software that captures the new real-world value and updates the database.  Consider that the act of recording the new value may or may not take place at the same time as the real-world value itself changed.  In other words the external event is distinct from the internal event.

I believe it is important to differentiate between external and internal events when developing a data model that captures the history of data states.  For one, recording internal events can help to audit data entry errors.  For example, erroneous data can sometimes be detected and even corrected if it is examined in the context of other temporally-proximal data entry events (e.g., a data entry operator that repeated a value from a previous data entry record).

Recording internal events explicitly can also help to troubleshoot querying anomalies caused by failure to take into account the difference in time between the occurrence of an external and its corresponding internal event.  For example, why did a query that was run at time t not reflect the updated state of the real world event that occurred at time t-1? It will not if the data entry--the internal event--occurred at time t+1.


Recording internal events can aid in the determining the time periods during which a database is "out of sync" with real-world entity values, due to data input errors.  Ideally, a database will always correctly reflect the state of the real-world, but in practice this is rarely the case. Inevitably, bad data will enter a system, and, at best, is corrected at some point in the future.  When corrections are made and recorded as internal events, the original and the corrected internal event timestamps can be used to determine when and for how long the database maintained inaccurate state.  If internal events can be marked as having been invalid, it is then also possible to generate reports that either ignore or include erroneous data states.  The advantage is that the database is not attempting to forget or otherwise hide the fact that erroneous data existed.  Much as database designers contend that data should never be deleted, one can argue that erroneous data should also not be deleted, but simply flagged.  Knowing that a database was temporarily maintaining bad data can be just as important as storing the correct values and their history.

Note that both external and internal events can also be used to record persons and comments associated with the event, in addition to just timestamps.

Internal events are most commonly recorded in log files, rather than as data in the database itself.  It can be very useful though to record internal events directly in the database, as this avoids the need to join log file output (of internal events) with database records of external events when performing troubleshooting or auditing tasks.

Configuring Spring-managed Hibernate event listeners with EntityManagerFactory

I'm in the process of migrating my work's web application from a pure Hibernate persistence architecture over to a JPA-based architecture (with Hibernate as the JPA implementation provider). The application uses Spring XML context files for its configuration. Previously, the persistence-related configuration defined a couple of custom Hibernate event listeners, as Spring-defined beans, and these were passed along to Spring's LocalSessionFactoryBean via its eventListeners property in a straightforward manner:

  <bean id="hibernateSessionFactory" class="org.springframework.orm.hibernate3.annotation.LocalSessionFactoryBean">
    ...
    <property name="eventListeners">
    <map>
      <entry key="post-load">
        <list>
          <bean ref="myPostLoadEventListenerBean" />
        </list>
    </map>
    </property>
  </bean>

However, after moving to a JPA-based configuration, and replacing the Hibernate-specific LocalSessionFactoryBean with a JPA-specific LocalContainerEntityManagerFactoryBean, the only way to declaratively specify the Hibernate event listeners is via the jpaPropertyMap. However, with this method, the event listeners can no longer be Spring-instantiated objects, but rather can only be specified as class names, which Hibernate will "conveniently" use to instantiate the event listener objects on its own. This is hardly convenient, and in the case where one's custom event listener classes rely upon Spring's dependency injection for initialization, Hibernate's initialization mechanism is downright limiting. What we would like to do, but cannot, is specify our event listener beans within the JPA-compliant jpaPropertyMap, as follows:

  <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    ...
    <property name="persistenceProvider">
      <bean class="org.hibernate.ejb.ConfigurableListenerBeansHibernatePersistence">
        <property name="postLoadEventListeners">
          <list>
            <bean ref="myPostLoadEventListener" />
          </list>
        </property>
      </bean>
    </property>
    <property name="jpaPropertyMap">
      <map>
         <entry key="hibernate.ejb.event.post-load">
           <list>
             <bean class="org.hibernate.ejb.event.EJB3PostLoadEventListener" />
             <bean ref="myPostLoadEventListener" />
           </list>
         </entry>
      </map>
    </property>
  </bean>

The limitation is ultimately introduced by Hibernate's EventListenerConfigurator, which only knows how to handle event listener class names, rather than event listener objects. That's fine, I suppose, but Spring should provide some factory class of its own that allows the configuration of JPA provider-specific properties (such as Hibernate's event listeners), to get around this limitation by taking the desired beans and then programmatically updating the underlying PersistenceProvider. But Spring does not provide this, as the LocalContainerEntityManagerFactoryBean simply passes along the jpaProperties without further consideration. In the end, Spring and Hibernate classes collude in a such a way that there is no way to accomplish what was easily done with the Hibernate-specific Spring configuration.

The solution I've come up with involves replacing Hibernate's HibernatePersistence class with an extended version of the class, ConfigurableListenerBeansHibernatePersistence, that specifically allows event listeners to be specified as objects, thus allowing Spring-instantiated bean to be injected. We then tell Spring's LocalContainerEntityManagerFactoryBean to use our special HibernatePersistence implementation, instead of the default HibernatePersistence class. Yes, this sidesteps the JPA-compliant interfaces, but I find no other way to accomplish the required injection of event listeners. The new Spring XML configuration looks like this:

  <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    ...
    <property name="persistenceProvider">
      <bean class="org.hibernate.ejb.ConfigurableListenerBeansHibernatePersistence">
        <property name="postLoadEventListeners">
          <list>
            <bean ref="myPostLoadEventListener" />
          </list>
        </property>
      </bean>
    </property>
    <property name="jpaPropertyMap">
      <map>
      </map>
    </property>
  </bean>

Seems like Spring should provide a similar solution out-of-the-box.

Note that ConfigurableListenerBeansHibernatePersistence also takes it upon itself to maintain the default event listeners that Hibernate normally creates on its own (Ejb3*EventListeners and various Default*EventListeners), and appends the application-provided listeners to these. Contrast this with the default configuration behavior, which requires a developer to specify the Hibernate-provided event listeners in addition to the listeners being added by the application. (Hibernate's approach is the most flexible, but a pain for developers that simply want to add new listeners that do not alter the underlying behavior of Hibernate.)

Finally, note that current version of ConfigurableListenerBeansHibernatePersistence only provides configurability of the PostLoadEvent for now.

Two UNIX tips

Picked up two UNIX tips at http://www.ffmpeg.org/faq.html#SEC27, which I'd like to remember:

• {tail,head} commands take a {+,-} modifier on their -n and -c options, which causes the command to skip the {leading,trailing} N lines or bytes of the input.
• A bash "compound command" that is defined using flanking "{" and "}" characters can be used to apply the backgrounding operator to a list of commands

The act of committing

The act of committing a new revision of one's code to a source control repository seems like a rather coarse operation.  When a developer makes a commit, he is in a way declaring that his code is "ready" for others to view or use.  If he commits to a feature branch or experimental branch, he is declaring that his code is ready at some initial level, but perhaps not ready for integration with a project's trunk or a team-wide release branch.  When a project trunk or release branch is declared ready for use, it is often "committed" to a new branch or tag, declaring its status as high-quality, tested code.

At the other end of the version control spectrum, developers often have access to a "local revision history" (e.g., as provide by Eclipse), where every save of a source file is tracked locally, and can be compared to other local revisions and may be reverted to an earlier revision as required.

With distributed version control systems, we have an intermediate level of committing, whereby commits are made to a local, privately-managed repository that can be optionally shared with others.  Such version control systems seem to encourage more frequent committing, as a means of safely recording one's development progress, without necessarily making one's efforts immediately public.

So I feel like we have a discrete levels of versioning functionality, which suggests the possibility of an alternate versioning model based on a more continuous spectrum of commit actions.

What if every change made to source code is tracked by the developer's versioning system?  And then, what if, at explicit times, a developer can choose to declare his code as being at a particular level of quality.  For example, while developing a new feature, the developer might be able to declare his code as "compiles", and later, "passes tests", and later still "functional", then "beta", "Q/A tested", and finally "released" (perhaps set by a release engineer), etc.  The idea here is that we don't force the developer into a single commit/no-commit decision point.  Instead, the developer can communicate the level of readiness of his code as it evolves.  Different levels of readiness can be configured to be kept private for different audiences.  So, for example, if the developer is working closely on a feature with one other developer, the two can see each others changes' at any point, or may chose to see only the changes that are "compilable", etc.  Different subteams on the same project may only want to be exposed to code that "passes tests", while developers on other projects that depend upon the code in this project may only want "Q/A tested" or "released" versions.

Such a versioning model would allow maximal sharing of code to appropriate audiences, while ensuring that the desired level of code quality/readiness is made available.  Developers wouldn't have to "call over the cubical wall" to ask his fellow developers if they have committed a particular change yet.  A developer could choose to see even the most ill-prepared code and keep tabs on it as it progresses, or even jump in and help out with edits on code that has just been entered by a teammate.  This has interesting implications for (non-co-located) team members that want to review each other's work or even engage in remote pair-program activities.

But the fundamental idea here is to increase the granularity of "commits", such that developers do not keep their code inaccessible from others for too long, while at the same time, preventing low-quality code from being exposed to users that would be harmed by it.  This seems like a safer alternative to committing infrequently, as it allows developers to not worry about publicizing their work that may not be ready for prime time.  Such a version control system would become a combination of "local revision history", "distributed version control", and "public commits" (to a central server).  It would encourage  developers to be more explicit about publicly declaring the state that a particular revision of their code is at.  It would avoid the "ready" versus "not ready" binary proposition that is forced upon us when using the commit model of current version control systems.  And it would maximize visibility and sharing of code that is under active development.