To do this, I drew a bit of help from Enlive, which functions primarily as a templating library, but has a few features useful for screen-scraping. This was perhaps a bit of overkill, as I only ended up using one of it’s functions, html-resource, which takes an URL as input, and outputs an hash that nicely represents a web-page’s structure.

What I ended up is wordy, which at the moment can do a simple word-count frequency analysis on a given page. That is, it counts how often words used, filtering (if desired) on word length. In just a bit, I’ll get into some of the more interesting aspects of coding it  up, but first,  here is a simple use case.

Running the following in slime…
(graph-words "http://ethanjfast.com" 5 5 1)

Where the parameters correspond to:

• ethanjfast.com -> web page to look at
• 5 -> minimum length (letter count) of word for first anaylsis
• 5 -> minimum length of word for last anaylsis
• 1 -> the amount of word length to increment by between the first and last anaylsis

To make this a bit clearer, consider a different run:
(graph-words "http://ycombinator.posterous.com" 3 10 3)

Here wordy does three analyses, with minimum word lengths of 3, 6, and 9 respectively. Clearly, I have some work to do insofar as these graphs look rather pathetic, but it was nice to get incanter working.

Now, onto some implementation details. Most of the code is quite simple, so I’ll just go through a few functions that may have some value to someone learning Clojure. For instance, here is rec-map, a function which recursively traverses the map/list structure returned by html-resource.

Basically, this function filters out all page content that doesn’t match specific tags (getting rid of links, css, javascript, ect.) But at first glance, you might wonder why I used trampoline rather than recur. After all, trampoline is used to recurse between two different functions, and it looks very much like rec-map is calling itself. Well, the trick is that I am calling trampoline inside the function passed to map, so recur will fail spectacularly (and in a very confusing manner). So watch out for recursion within anonymous functions!

Here is another bit of code, where I create the graph with Incanter.

The :group-by parameter is slightly unintuitive. To use it, you make a new vector of labels, each label mapping to a counterpart in the data vector. All data with the same label are then put into the same group (e.g for data ["You" "Me" "I"] [3 2 4] one might use the label vector [0 1 1] to group “Me” and “I” together). The rest is fairly self-explanatory, but I’ll mention one thing that I didn’t know until this morning. You can’t nest the # function shortcut. For instance, the following would not work:

(map #(map #(first %1) %1) lst)

It’s rather obvious in retrospect, I know. But I was dumb enough to try it. That’s all for now, and the code is available on github.

In any case, there was a recent small flurry of activity regarding Paul Graham’s use of the rhetorical device “it turns out.” Not to put too fine a point on it, but it turns out that these claims of hacks and benign disingenuity amount to something so small that I would call it nothing (albeit, a very clever nothing).

Who am I to say such a thing? Well, I have data, lovingly provided by the labors or a small ruby script and the Hpricot gem. By my count, most of pg’s statements do not require any great degree of rhetorical aid, as implied by this analysis (which, despite my disagreement, is excellently written). Take a look for yourself (and pardon the potentially regex-induced typos):

From: http://www.paulgraham.com/angelinvesting.html
When we sold our startup in 1998 I thought one day I’d do some angel investing. Seven years later I still hadn’t started. I put it off because it seemed mysterious and complicated. It turns out to be easier than I expected, and also more interesting.The part I thought was hard, the mechanics of investing, really isn’t. You give a startup money and they give you stock. You’ll probably get either preferred stock, which means stock with extra rights like getting your money back fir…

From: http://www.paulgraham.com/hackernews.html
…g ideas in the future, and the ones that turn out to work will probably seem just as broken as those that don’t.Probably the most important thing I’ve learned about dilution is that it’s measured more in behavior than users. It’s bad behavior you want to keep out more than bad people. User behavior turns out to be surprisingly malleable. If people are expected to behave well, they tend to; and vice versa.Though of course forbidding bad behavior does tend to keep away bad people, because they feel uncomfortably constrained in a place where they have to behave well. But this way of keeping the…

From: http://www.paulgraham.com/highres.html
…mously successful organizations like the Roman army or the British East India Company were any less afflicted by protocol and politics than organizations of the same size today. But they were competing against opponents who couldn’t change the rules on the fly by discovering new technology. Now it turns out the rule “large and disciplined organizations win” needs to have a qualification appended: “at games that change slowly.” No one knew till change reached a sufficient speed.Large organizations will start to do worse now, though, because for the first time in history they’re no longer gettin…

From: http://www.paulgraham.com/cities.html
… It’s probably the place in America where someone from Northern Europe would feel most at home. But it’s not humming with ambition.In retrospect it shouldn’t have been surprising that a place so pleasant would attract people interested above all in quality of life. Cambridge with good weather, it turns out, is not Cambridge. The people you find in Cambridge are not there by accident. You have to make sacrifices to live there. It’s expensive and somewhat grubby, and the weather’s often bad. So the kind of people you find in Cambridge are the kind of people who want to live where the smartes…

From: http://www.paulgraham.com/distraction.html
…owse the web. (Irony of ironies, it’s the computer Steve Huffman wrote Reddit on. When Steve and Alexis auctioned off their old laptops for charity, I bought them for the Y Combinator museum.)My rule is that I can spend as much time online as I want, as long as I do it on that computer. And this turns out to be enough. When I have to sit on the other side of the room to check email or browse the web, I become much more aware of it. Sufficiently aware, in my case at least, that it’s hard to spend more than about an hour a day online.And my main computer is now freed for work. If you try th…

From: http://www.paulgraham.com/good.html
… truth you don’t have to remember anything, and that’s a really useful property in domains where things happen fast.For example, Y Combinator has now invested in 80 startups, 57 of which are still alive. (The rest have died or merged or been acquired.) When you’re trying to advise 57 startups, it turns out you have to have a stateless algorithm. You can’t have ulterior motives when you have 57 things going on at once, because you can’t remember them. So our rule is just to do whatever’s best for the founders. Not because we’re particularly benevolent, but because it’s the only algorithm th…

From: http://www.paulgraham.com/philosophy.html
…s examples of how to argue: they hoped they were getting results. Most were wrong, but it doesn’t seem an impossible hope.This argument seems to me like someone in 1500 looking at the lack of results achieved by alchemy and saying its value was as a process. No, they were going about it wrong. It turns out it is possible to transmute lead into gold (though not economically at current energy prices), but the route to that knowledge was to backtrack and try another approach.Thanks to Trevor Blackwell, Paul Buchheit, Jessica Livingston, Robert Morris, Mark Nitzberg, and Peter Norvig for reading…

From: http://www.paulgraham.com/colleges.html
…is was the new trend of worrying obsessively about what kindergarten your kids go to. It seemed to me this couldn’t possibly matter. Either it won’t help your kid get into Harvard, or if it does, getting into Harvard won’t mean much anymore. And then I thought: how much does it mean even now?It turns out I have a lot of data about that. My three partners and I run a seed stage investment firm called Y Combinator. We invest when the company is just a couple guys and an idea. The idea doesn’t matter much; it will change anyway. Most of our decision is based on the founders. The average …

From: http://www.paulgraham.com/head.html
…t possible.Probably the best we’ll do is some kind of hack, like making the programming parts of an organization work differently from the rest. Perhaps the optimal solution is for big companies not even to try to develop ideas in house, but simply to buy them. But regardless of what the solution turns out to be, the first step is to realize there’s a problem. There is a contradiction in the very phrase “software company.” The two words are pulling in opposite directions. Any good programmer in a large organization is going to be at odds with it, because organizations are designed to preve…

From: http://www.paulgraham.com/wisdom.html
…adicts them.Is the mathematician a small man because he’s discontented? No; he’s just doing a kind of work that wasn’t very common in Confucius’s day.Human knowledge seems to grow fractally. Time after time, something that seemed a small and uninteresting area—experimental error, even—turns out, when examined up close, to have as much in it as all knowledge up to that point. Several of the fractal buds that have exploded since ancient times involve inventing and discovering new things. Math, for example, used to be something a handful of people did part-time. Now it’s the caree…

From: http://www.paulgraham.com/goodart.html
…itional//EN”> How Art Can Be Good December 2006I grew up believing that taste is just a matter of personal preference. Each person has things they like, but no one’s preferences are any better than anyone else’s. There is no such thing as good taste.Like a lot of things I grew up believing, this turns out to be false, and I’m going to try to explain why.One problem with saying there’s no such thing as good taste is that it also means there’s no such thing as good art. If there were good art, then people who liked it would have better taste than people who didn’t. So if you discard taste, y…

From: http://www.paulgraham.com/mit.html
…personal bias points in the same direction technology evolves in.The advantages of rootlessness are similar to those of poverty. When you’re young you’re more mobile—not just because you don’t have a house or much stuff, but also because you’re less likely to have serious relationships. This turns out to be important, because a lot of startups involve someone moving.The founders of Kiko, for example, are now en route to the Bay Area to start their next startup. It’s a better place for what they want to do. And it was easy for them to decide to go, because neither as far as I know has a…

From: http://www.paulgraham.com/marginal.html
… comics might seem to the average person today.In the computer world we get not new mediums but new platforms: the minicomputer, the microprocessor, the web-based application. At first they’re always dismissed as being unsuitable for real work. And yet someone always decides to try anyway, and it turns out you can do more than anyone expected. So in the future when you hear people say of a new platform: yeah, it’s popular and cheap, but not ready yet for real work, jump on it.As well as being more comfortable working on established lines, insiders generally have a vested interest in perpetua…

From: http://www.paulgraham.com/america.html
…upation– which is not far from the idea that each person has a natural “station” in life. If this were true, the most efficient plan would be to discover each person’s station as early as possible, so they could receive the training appropriate to it.In the US things are more haphazard. But that turns out to be an advantage as an economy gets more liquid, just as dynamic typing turns out to work better than static for ill-defined problems. This is particularly true with startups. “Startup founder” is not the sort of career a high school student would choose. If you ask at that age, people…

From: http://www.paulgraham.com/randomness.html
…e would. We’d ask why we even suppose we have a “purpose” in life. We may be better adapted for some things than others; we may be happier doing things we’re adapted for; but why assume purpose?The history of ideas is a history of gradually discarding the assumption that it’s all about us. No, it turns out, the earth is not the center of the universe—not even the center of the solar system. No, it turns out, humans are not created by God in his own image; they’re just one species among many, descended not merely from apes, but from microorganisms. Even the concept of “me” turns out to…

From: http://www.paulgraham.com/startupfunding.html
… a loan that can be converted into stock later; it works out the same as a stock purchase in the end, but gives the angel more protection against being squashed by VCs in future rounds.Who pays the legal bills for this deal? The startup, remember, only has a couple thousand left. In practice this turns out to be a sticky problem that usually gets solved in some improvised way. Maybe the startup can find lawyers who will do it cheaply in the hope of future work if the startup succeeds. Maybe someone has a lawyer friend. Maybe the angel pays for his lawyer to represent both sides. (Make sure…

From: http://www.paulgraham.com/ideas.html
…enomenon when I worked on spam filters. In 2002, most people preferred to ignore spam, and most of those who didn’t preferred to believe the heuristic filters then available were the best you could do.I found spam intolerable, and I felt it had to be possible to recognize it statistically. And it turns out that was all you needed to solve the problem. The algorithm I used was ridiculously simple. Anyone who’d really tried to solve the problem would have found it. It was just that no one had really tried to solve the problem. [3]Let me repeat that recipe: finding the problem intolerable a…

From: http://www.paulgraham.com/inequality.html
… confiscate whatever you deem to be surplus.So let’s be clear what reducing economic inequality means. It is identical with taking money from the rich.When you transform a mathematical expression into another form, you often notice new things. So it is in this case. Taking money from the rich turns out to have consequences one might not foresee when one phrases the same idea in terms of “reducing inequality.”The problem is, risk and reward have to be proportionate. A bet with only a 10% chance of winning has to pay more than one with a 50% chance of winning, or no one will take it. So…

From: http://www.paulgraham.com/opensource.html
…can I claim business has to learn it? When I say business doesn’t know this, I mean the structure of business doesn’t reflect it.Business still reflects an older model, exemplified by the French word for working: travailler. It has an English cousin, travail, and what it means is torture. [2]This turns out not to be the last word on work, however. As societies get richer, they learn something about work that’s a lot like what they learn about diet. We know now that the healthiest diet is the one our peasant ancestors were forced to eat because they were poor. Like rich food, idleness only s…

From: http://www.paulgraham.com/submarine.html
… came from. I could tell a lot of them were crap, but I didn’t realize why.Remember the exercises in critical reading you did in school, where you had to look at a piece of writing and step back and ask whether the author was telling the whole truth? If you really want to be a critical reader, it turns out you have to step back one step further, and ask not just whether the author is telling the truth, but why he’s writing about this subject at all.Online, the answer tends to be a lot simpler. Most people who publish online write what they write for the simple reason that they want to. You …

From: http://www.paulgraham.com/bronze.html
… a field like that would be dominated by fearsome startups with five million dollars of VC money each. Whereas we felt pretty sure that we could hold our own in the slightly less competitive business of generating Web sites for art galleries.We erred ridiculously far on the side of safety. As it turns out, VC-backed startups are not that fearsome. They’re too busy trying to spend all that money to get software written. In 1995, the e-commerce business was very competitive as measured in press releases, but not as measured in software. And really it never was. The big fish like Open Mark…

From: http://www.paulgraham.com/venturecapital.html
…rcentage of the gains. So they want the fund to be huge– hundreds of millions of dollars, if possible. But that means each partner ends up being responsible for investing a lot of money. And since one person can only manage so many deals, each deal has to be for multiple millions of dollars.This turns out to explain nearly all the characteristics of VCs that founders hate.It explains why VCs take so agonizingly long to make up their minds, and why their due diligence feels like a body cavity search. [2] With so much at stake, they have to be paranoid.It explains why they steal your ideas. E…

From: http://www.paulgraham.com/start.html
…corporating it, of course: insurance, business license, unemployment compensation, various things with the IRS. I’m not even sure what the list is, because we, ah, skipped all that. When we got real funding near the end of 1996, we hired a great CFO, who fixed everything retroactively. It turns out that no one comes and arrests you if you don’t do everything you’re supposed to when starting a company. And a good thing too, or a lot of startups would never get started. [5]It can be dangerous to delay turning yourself into a company, because one or more of the founders might decide to s…

From: http://www.paulgraham.com/hs.html
…s to learn what the options were. You don’t need to be in a rush to choose your life’s work. What you need to do is discover what you like. You have to work on stuff you like if you want to be good at what you do.It might seem that nothing would be easier than deciding what you like, but it turns out to be hard, partly because it’s hard to get an accurate picture of most jobs. Being a doctor is not the way it’s portrayed on TV. Fortunately you can also watch real doctors, by volunteering in hospitals. [1]But there are other jobs you can’t learn about, because no one is doing them yet….

From: http://www.paulgraham.com/laundry.html
…turned against them by clumsy, self-appointed tour guides. The other big difference between a real essay and the things they make you write in school is that a real essay doesn’t take a position and then defend it. That principle, like the idea that we ought to be writing about literature, turns out to be another intellectual hangover of long forgotten origins. It’s often mistakenly believed that medieval universities were mostly seminaries. In fact they were more law schools. And at least in our tradition lawyers are advocates: they are trained to be able to take either side of an …

From: http://www.paulgraham.com/bubble.html
…ing happened during the Mississippi and South Sea Bubbles. What drove them was the invention of organized public finance (the South Sea Company, despite its name, was really a competitor of the Bank of England). And that did turn out to be a big deal, in the long run.Recognizing an important trend turns out to be easier than figuring out how to profit from it. The mistake investors always seem to make is to take the trend too literally. Since the Internet was the big new thing, investors supposed that the more Internettish the company, the better. Hence such parodies as Pets.Com.In fact mos…

From: http://www.paulgraham.com/essay.html
…was, 700 years ago, fascinating and urgently needed work.No DefenseThe other big difference between a real essay and the things they make you write in school is that a real essay doesn’t take a position and then defend it. That principle, like the idea that we ought to be writing about literature, turns out to be another intellectual hangover of long forgotten origins.It’s often mistakenly believed that medieval universities were mostly seminaries. In fact they were more law schools. And at least in our tradition lawyers are advocates, trained to take either side of an argument and make as g…

From: http://www.paulgraham.com/gh.html
…ket. What VCs should be looking for is the next Apple, or the next Google.I think Bill Gates knows this. What worries him about Google is not the power of their brand, but the fact that they have better hackers. [7] RecognitionSo who are the great hackers? How do you know when you meet one? That turns out to be very hard. Even hackers can’t tell. I’m pretty sure now that my friend Trevor Blackwell is a great hacker. You may have read on Slashdot how he made his own Segway. The remarkable thing about this project was that he wrote all the software in one day (in Python, incidentally).For …

From: http://www.paulgraham.com/wealth.html
…he difficulty of assigning a value to each person’s work. For the most part they punt. In a big company you get paid a fairly predictable salary for working fairly hard. You’re expected not to be obviously incompetent or lazy, but you’re not expected to devote your whole life to your work.It turns out, though, that there are economies of scale in how much of your life you devote to your work. In the right kind of business, someone who really devoted himself to work could generate ten or even a hundred times as much wealth as an average employee. A programmer, for example, instead of …

From: http://www.paulgraham.com/gba.html
…pt countries become poor; and poor countries are weak. It seems to me there is a Laffer curve for government power, just as for tax revenues. At least, it seems likely enough that it would be stupid to try the experiment and find out. Unlike high tax rates, you can’t repeal totalitarianism if it turns out to be a mistake.This is why hackers worry. The government spying on people doesn’t literally make programmers write worse code. It just leads eventually to a world in which bad ideas win. And because this is so important to hackers, they’re especially sensitive to it. They can sense tot…

From: http://www.paulgraham.com/hp.html
…eers figure out how to do it.What and how should not be kept too separate. You’re asking for trouble if you try to decide what to do without understanding how to do it. But hacking can certainly be more than just deciding how to implement some spec. At its best, it’s creating the spec– though it turns out the best way to do that is to implement it.Perhaps one day “computer science” will, like Yugoslavia, get broken up into its component parts. That might be a good thing. Especially if it meant independence for my native land, hacking.Bundling all these different types of work together in o…

From: http://www.paulgraham.com/spam.html
…l analysis, I found immediately that it was much cleverer than I had been. It discovered, of course, that terms like “virtumundo” and “teens” were good indicators of spam. But it also discovered that “per” and “FL” and “ff0000″ are good indicators of spam. In fact, “ff0000″ (html for bright red) turns out to be as good an indicator of spam as any pornographic term._ _ _Here’s a sketch of how I do statistical filtering. I start with one corpus of spam and one of nonspam mail. At the moment each one has about 4000 messages in it. I scan the entire text, including headers and embedded html…

From: http://www.paulgraham.com/icad.html
…at also means there will always be lots of Java programmers, so if the programmers working for me now quit, as programmers working for me mysteriously always do, I can easily replace them.Well, this doesn’t sound that unreasonable. But it’s all based on one unspoken assumption, and that assumption turns out to be false. The pointy-haired boss believes that all programming languages are pretty much equivalent. If that were true, he would be right on target. If languages are all equivalent, sure, use whatever language everyone else is using.But all languages are not equivalent, and I think I …

From: http://www.paulgraham.com/road.html
…an of any loose objects that might later get stuck in something.It helps if you use a technique called functional programming. Functional programming means avoiding side-effects. It’s something you’re more likely to see in research papers than commercial software, but for Web-based applications it turns out to be really useful. It’s hard to write entire programs as purely functional code, but you can write substantial chunks this way. It makes those parts of your software easier to test, because they have no state, and that is very convenient in a situation where you are constantly making an…

From: http://www.paulgraham.com/javacover.html
…ught customers would want, or something they were told to do by management. These are smart people; if the technology was good, they’d have used it voluntarily.6. It has too many cooks. The best programming languages have been developed by small groups. Java seems to be run by a committee. If it turns out to be a good language, it will be the first time in history that a committee has designed a good language.7. It’s bureaucratic. From what little I know about Java, there seem to be a lot of protocols for doing things. Really good languages aren’t like that. They let you do what you want …

Now, what kind of runner must bear this hideous burden of proof? To be presumed crazy before sane, and intoxicated before clean? What kind of runner is dragged off his joyous release of energy on the streets and trails of his neighborhood? Well, apparently a barefoot one.

Granted, these policemen – these unwanted and would-be protectors of my person — they did have something going for them. It was 35 degrees outside. Snow lay piled in banks along the sidewalks and road-corners. Yet there I was, this strange young man in running shorts and a sweatshirt, leaping over and atop piles of snow, padding gracefully down ice-encrusted sidewalks. Without shoes. It must have been clear to them, and to the logic of their misplaced assumptions: surely, no sane person would be engaging in such pursuits.

Well, they were wrong. Their premises were wrong. This did not, of course, prevent them from taking control of the situation. Alas, it was a tragedy in which all parties were confused. At first, I thought the officers were being friendly in that ignorant way with which I am so intimately familiar. I’ve been offered rides by many strangers, and apparently the local assumption is that I’m some kind of poor lost kitten, in need of help on his journey home.

So as they pulled over and cracked the door of their police car, I smiled and waved them off. “No, I’m fine,” I said. “I’m running!” In retrospect, they did seem more determined then one’s average do-gooder, but to my peril, I did not notice this at the time.

“Wait,” they called.

“I’m fine,” I repeated. As I passed away and forward, they shouted once more at me to come back, and I miscalculated by ignoring this as well — by speeding up, in fact.

So it was that I entered the short — and as things would end up, final — stretch of my run. A bit of a race with a police car, as it were, although strictly speaking this was not really intended. Stupidly, I thought the officers would give up their offers of uninformed aid when they noticed my obvious competence (I can go fast, you see) and unambiguous obstinacy. But I, too, was operating under misassumptions. These officers were not playing in the mental zone of the good samaritan, but rather that more enticing arena of catch and subdue the crazy person.

In the end there was no subduing. Upon a second rendezvous, the motives of the officers were quite apparent, and I quickly complied, although only against the risks and uncertainties of my alternatives. In the end, what I needed was someone to vouch for me, a duty that my roommate happily performed. And so it was that the exalted release which I normally gain from running transformed into a proverbial slap in the face, a gift from those friends of misunderstood culture.

But this was not all. I was provided with a card that I would do well to keep with me upon all my runs, such that I might “prove myself” to other officers. The compassion! Inexperienced as I am, I’ll offer this (half-serious) warning to cold weather barefoot runners: you are being hunted; travel in packs.

Most people would agree, I think, that there are many ways to succeed, and a nearly infinite number of things upon which such success can be centered. But what does this mean? After all, a successful father is quite different from a successful businessmen, and yet “success” can be used, quite appropriately, to describe each. This is not a hard question; at first brush, the answer seems obvious. In each instance, a goal was set — meaningful fatherhood, or growing a business – and likewise, in each instance the particulars of a goal were achieved.

But is that all there is to success, the fulfillment of goals? It seems unlikely that an incarcerated man would consider the fulfilling of his imprisonment a successful endeavor. Clearly, there must be a subjective side to the term. To be successful at something, one must desire it. So success requires at least two things: a goal and a desire. Take either away, and you destroy its meaning. Consider the hypothetical beneficiary of vast and unexpected charity. She had no pans to acquire wealth, and so she is no more successful than our aforementioned prisoner, even if she counted money among her desires. Without a goal in place, its achievement means nothing.

Now, all this is quite obvious and boring; one might suggest that it’s just a quick and poor deconstruction of semantics. The real and practical question is whether all unique instances of success stem from any common source. Effectively, one might ask, among all the disparate successes of the world, do they have anything in common? Can the businessman and the father — at the most general of levels — share a common game plan? I would argue that they can, and that they do. Moreover, I think this common element is initiative.

Certainly, when starting a business, initiative plays an integral role. Every company in present existence required, at minimum, the initiative of creation, the actions of a prime mover who would set it up to succeed (or fail), as history will dictate. But more than this, there is a broader sense in which initiative is tied to the founding of companies. Businesses are started, and markets created, to meet the needs of a particular consumer base. Identifying that need, or alternatively, improving upon the realization of an existing need, by definition requires initiative. It necessitates a seeing thorough of reality, a questioning of what is, and the recognition of a world that might be.

Success in science, too, is bound up in initiative. Scientific research explores the undiscovered territory of human knowledge. It takes initiative to make that first step into the dark, to decide where to place one’s efforts, and to question and improve upon the current state of knowledge. Moreover, the grand revolutions of science are steeped in such character. I doubt it needs mentioning, but Einstein’s discovery of special relativity occurred while he was a patent clerk — that’s clearly initiative, if ever any has existed.

But what about the father, and those softer definitions of success? Here particularly, my argument can be seen to break down on some boundary cases. Take, for example, a rich and carefree fellow who has as his “fatherly” desire and goal: my children shall not starve. Well, even though by his own definition he is successful, it’s hard to see such a person as displaying initiative. But this, I think, is a special case. For most people, in most circumstances, the goals by which they characterize success are achieved by standards outside of their personal norm. And almost by definition, if a goal is difficult to achieve, then one might benefit from the application of initiative. After all, if it’s easy to acquire something without any initiative whatsoever, you probably have it already.

So, one might say I’m rather a fan of initiative — at least, as far as achieving success is concerned.  Yet strangely, most people that I talk to don’t hold the same opinion. I’ll ask them how they think most people succede, and the answers are usually intelligence, dedication, or perhaps, luck. All these things surely play a role in success, but, in my humble view, they are not the prime mover of successful endeavors — no, that role is played by initiative.

1. Identify problem
2. On the grounds of theoretical purity, construct needlessly complicated solution.
3. Implement solution.

For instance, a few days ago I wanted to put together a small rails application. There was nothing earth shattering in this; I simply wanted to get a quick prototype up and running. So I update my gems, do a quick `rails prototype/` command, and boot up the empty app. It fails.

Ok, so I do the natural thing; I google the error message or whatnot; I search for a few minutes. Well, it has something to do with Snow Leopard, that much becomes clear. This begs the question: does any bug ever not have something to do with Snow Leopard? But I won’t adress that here. Anyhow, there is an obscure solution, under which I’d have to edit manually my rails installation — something like that, anyway.  But I’m not going to do that. I try some earlier versions of rails and it’s dependancies — they fail for some other reasons.

I’m sure that I could have figured it out, had I dug around for a while. I might have, with work, discovered the vagaries behind why the same programs, with the same (nominally) installed dependancies exhibited opposite behaviors on different machines. But no, by next month I’d probably just have a new and similar problem. So I do just what any irrational fellow ought to; I decide to install a virtual Arch distro (my favorite) with VMware fusion, mount the machine as a disk via ssh, and develop from there. At least in Arch, I will know exactly what is installed, where it is, and how it ought to be working. Basically, I eliminate the free variable of imprecise ignorance regarding everything that’s on the machine, and whether I’ve hacked it up manually. Of course, god forbid that I give up TextMate and actually code directly on a linux machine. Emacs is awesome for my more lispy projects, but I like TM for rails.

To be clear, the blame here is almost entirely on me. However, given the recent announcement of  the iPad, I feel that it’s become the custom to lay down some hate. So here goes: Apple, you ought to implement a decent package management scheme. Some inconsistency with something I had installed (or worse, you put there) was messing things up. Sure, one might say that I use pacman (or yum, or insert-reader-favorite) as a crutch, and I’m fine with that. In particular, said management helps prevent people like me — overly exuberant installers and occasional hackers of who-knows-what – from messing up systems.

So on the one hand, I get what I want. Things tend to work on Arch, and updating is only a `pacman -Syu` away. On the other hand — yeah, I’m dumb.

As always, the code open source and available at github.

For instance, although the function run-ga only takes two parameters, both of them are hashes. Naturally, each must contain a few keys for the genetic algorithm to run properly. Using preconditions, it’s easy to ensure that the function’s parameters contain these required keys. For instance:

(defn keys-not-nil [lst hash]
  (reduce #(and %1 %2) (map #(not (nil? (hash %))) lst)))

This checks each key in a list of keys, and returns false if one or more of these keys maps to a value of nil. Obviously, I could go further here, but checking for non-nil values seemed a reasonable (and easy) generalization (e.g. if :init-fn or :mut-r map to nil, then the ga-run function will have a problem). Now, making use of the :pre tag.

(defn run-ga
[func-map setting-map]
{:pre [(and
  (keys-not-nil
    (list :init-fn :fit-fn :mut-fn :sel-fn :cross-fn)
    func-map)
  (keys-not-nil
    (list :pop-sz :gen :children :mut-r)
    setting-map))]}
... actual algorithm ...)

So I can now enforce the (arbitrary) required keys, and return assertion errors on some improper uses of ga-run. For a more specific example, look to the algorithm’s mutation function.

(defn generic-mutation
  "Randomly mutates lists with elements from other lists in the population."
  [list prob]
  {:pre [(and (>= prob 1) (<= prob 100))]
   :post [(list? %)]}
  (map
   (fn [s-list]
     (map
      (fn [test]
        (if (> prob (rand-int 100))
          (let [r-s (rand-int (count list))
                r-t (rand-int (count (nth list r-s)))]
            (nth (nth list r-s)
                 r-t))
          test))
      s-list))
   list))

Here, the :pre tag ensures that the value of prob is between one and one hundred. Similarly, :post asserts that the function returns a list (although I haven’t mentioned it, the :post tag operates much like :pre, using a function’s return value within its assertion). So in spite of my trivial examples, perhaps you can begin to appreciate how these tags can be used to create safer and more predictable code.

Now, I doubt that it will come as a surprise, but I am rather a newbie at this whole R thing, and as I later found out, loops in R should be avoided at all costs. After hacking around with my code, I found that its critical path looked something like this:

total <- 0
for(i in 1:nrow(dataset)){
total <- total + sum( dataset[i,1:25]*coef )
}

Now this is very simple loop, and it seemed to me somewhat less than obvious that it would beget a significant performance bottleneck. Ever so naturally, then, it did.

Ironically, the solution here is to use code more along the lines of the map-reduce paradigm, something I would have loved to do in the first place, were not I overcome by the cryptic nature of R’s documentation. After all, my favorite languages are all variants of lisp, and I am no stranger to functional programming. After some digging, I stumbled across apply, which more-or-less functions along the lines of map in scheme or clojure. So I tried:

my_sum <- function(x){ sum( x[1:25]*coef ) }
sum( apply( dataset, my_sum ) )

In addition to being more elegant, this is much, much faster. What was taking hours, now takes tens of seconds. Apparently, R has a fast backend implementation for this sort of thing.  So, this post is dedicated to as a warning to my fellow inexperienced users: avoid iterative loops in R!

As this blog serves a largely technical purpose, I ought to mention my use of latex (specifically, LaTeXIT) for all of November’s writings. This decision was mostly to my benefit — the pdf files produced look wonderful, and the memoir package made chapter organization easy — but it also beget a few annoyances. For instance, getting decent word count updates was rather non-trivial, and the NaNoWriMo site does not accept pdf or tex files for verification. Ultimately, however, my experience was a positive one, and I would recommend latex to future participants.

In any case, I can assure the nonexistent reader that this blog should become more active in future days and weeks.