Thursday, March 25, 2010

Quick Thoughts on DuckDuckGo

DuckDuckGo is a search engine. It is a Google competitor. Which also makes it a Bing competitor. Currently, it is a better decision-engine than Bing, and it's rather more useful than Google for certain search terms.

Someone once pointed out that if a search engine were to be built on good design, with a focus on making search results easier to absorb, they'd be able to compete with Google on a more-or-less even scale. DuckDuckGo seems to be doing just that, and I think they're doing it pretty well.

The most amazing thing about it, however, has been that it's all self-funded. And the dude's even responding to feature requests on Hacker News. (He implies that the entire search engine's a one-man show - by that I mean that he says "[Duck Duck Go] is a complicated mashup of my (emphasis added) index/crawler, structured crawls/dumps from crowd-source sites, live vertical APIs, and highly modified BOSS (edited, re-ranked, merged, omitted, etc.))

I think an acquisition seems likely. Still, gotta admire the balls on this entrepreneur for taking Google head-on.

Update: DuckDuckGo's results page for CS3216. Never knew CS3216 was a chemical compound! Hrmm!

Update 2: See searches for Java, Microsoft, and iPad.

Wednesday, March 24, 2010

Work Work

Henry graciously offered to give us a lift back to our residences, and so Adhi and I are in the car, talking.

"You know the irony is that the weekends are less free than the weekdays?" Adhi is saying, waving his hands in the front seat, "On the weekends we have to do this, but on the weekdays it's less busy because we have other modules to worry about. Now what kind of twisted logic is that?"

I murmur my agreement. We had just spend the evening discussing our app's implementation and there was still a lot of work to do.

"Sometimes I wish we had extra hours in the day." Adhi continues. I see Henry smiling in the rear view mirror as he turns the car past the Mochtar Riady building - we're only minutes away from the bus terminal, now.

"But why just extra hours?" I say. "You know, there's this comic called Dragon Ball, and there used to be a room in the comic where one minute is one hour, or something like that. I sometimes wonder what it'd be like to have that room - imagine preparing for your exams 20 minutes before the paper, or even 5 minutes ..."

"Sure from Dragon Ball?" Henry asks. "Sounds like something from siaw ting tang."

"Yeah sure." I say, and then to Adhiraj: "But why would you want extra hours? Why have hours when you can have days?"

"Oh no, that won't work." Adhi says, "We'll still have only two free days per week."

"What? Wait ... why?"

Adhi turns round to look at me. "Because NUS would take away the extra days from us." he says, in all seriousness. "They'll take that for themselves and just leave us with the two normal weekend days to relax."

I pause.

"Damn, you're right."

"Uhhuh."

"You're absolutely right."

And so, with that disturbing thought in our heads, we left the car and walked to our rooms and got right back to work.

Tuesday, March 16, 2010

Meritocracy

I have a sit-in lab to prepare for, so I must keep this short. Two things.

First I must say that I loved Zit Seng's presentation on web performance, despite having a runny nose throughout a large part of it. While I didn't understand most of what he was talking about, I did, more or less, get the general feel of how hard keeping servers up and running actually are in the real world. I think it was halfway through the presentation that I turned around to ask Kah Hong if he had experience in any of this, and he said no, and I pointed out that every time sgBeat's server crashed his mum would be the one to get it up and running again. He chuckled at that.

(I must also say - as a Mac user in NUS - that Zit Seng's blog is an invaluable resource on how to connect, print, and pretty much configure your Mac to work with SoC/NUS related assets. I don't know how I could've survived my first sem without him. The guy is just plain brilliant.)

Second, I rather enjoyed Prof's session on "Grades, Scholarships and Meritocracy". I didn't have much to say throughout the lecture, simply because there wasn't much I disagreed with. (Well, it was either that or I didn't have much interest in how Singapore manages its scholarship allocation.) After the lecture Tomithy asked how I was going to respond to Prof's arguments, but the truth is that there isn't much argument to be had. Our discussion (of which the first few comments have disappeared, due to a bug in Blogger) was primarily centered around the causes for a declining importance in grades.

What struck me the most has been that this session even existed. 50 years ago the advice would've been: "get good grades and you get a good job. Period." Today the advice is: "grades are important only for your first job." Could there be another decrease, 50 years down the road? I am convinced that there would be, but to what I am not particularly sure. I believe it would be really interesting to find out.

One last point, on meritocracy.

I am a big supporter of meritocracy, because I believe it works. It is clear, too, that Singapore is one of the ideology's most fervent worshippers. But in order to play with meritocracy, in order to live with it, it would do to examine some of its larger flaws.

What is meritocracy, in its purest form? Meritocracy essentially says that if you have talent and drive, you would and should rise to the top. But it also implies that those who are at the bottom, deserve to be at the bottom, and should stay there so long as they deserve it. It implies that our failures are merited and deserved the same way our successes are merited and deserved.

In the middle ages, when you met a poor person, that person would be described as 'an unfortunate'. Literally - an unfortunate; a person who has not been blessed by fortune. Today - and particularly in meritocratic societies such as in the US and in Singapore - such a person might be more unkindly described as a 'loser'.[1]

Alain de Botton points to Emile Durkheim, whose work in sociology shows us that such thinking leads to increased rates of suicide. "There are more suicides in developed individualistic countries than in any other part of the world." he says, "And some of the reason for that is that people take what happens to them extremely personally. The own their success. But they also own their failure."

When people say that many Singaporeans are unhappy, we should recognize that this is a net effect of meritocracy in action. The members of a meritocratic society are told that they can be whatever they want so long as they have the talent and ability to do so. When the reality doesn't reflect this ideal, people become very rapidly anxious, and envious, and just largely unhappy with life.

Perhaps it would be less of a problem if the reality reflected the ideals proposed by meritocracy. But this cannot ever be true: the reality is that there are too many random factors: there are accidents of birth, illnesses, freak explosions, and so on so forth, to truly reward people according to their merit. I have a friend in my cluster who has probably more business sense than most of my cohort combined. He is a fourth year student; very interested in entrepreneurship, but he is honest with himself. His parents are unwell. He would probably have to work at a job in order to take care of them.

A somewhat related idea that we now hold today is that we are told that we - and only we - are in control of our destinies. This is ridiculous, of course. There is as much probability of you becoming the next Bill Gates as there is of a peasant farmer becoming royalty back in the 1800s, in England (Botton uses the example of the French aristocracy). The difference is that today it isn't made to feel that way. We are told that we can go out and do whatever, be whoever we want to be, because of meritocracy, and this isn't always true.

There was a paradox in last night's session, and I'm not sure if many of us noticed it. Prof talked about meritocracy in the third part of his lecture. But he also mentioned that life is inherently unfair. What we seem to forget is that the two ideas are diametrically opposed to each other. While meritocracy is a beautiful philosophy, we must also be honest about its failings - there is no such thing as a perfectly meritocratic society. It is impossible. We just like to think that it isn't.

Meritocracy is mankind's finger at the face of life's unfairness, and if we think about it carefully: life usually wins.

How you deal with all that unfairness is up to you. Historically speaking, mankind's solution has been to turn to religion. In religion you are told that you are not always responsible for your failures: that sometimes, due to accidents of birth, or luck, or happenstance, you fail and it is not your fault. That is the kinder view of things, and it is one that I subscribe to; there may be others, of course.

I am not suggesting that we scrap meritocracy altogether. I am a big believer in the ideals of meritocracy. I think it is the best thing we've got, barring all alternatives. I am however arguing that we must be honest with ourselves. Meritocracy isn't a perfect ideology, there are problems with it, and those problems are often what leads us to become a deeply unhappy people.

1. Adapted from Alain de Botton's TED talk.

Sunday, March 14, 2010

{Essay} Why Grades?

Cross posted from my personal blog. The {Essay} tag that I use denotes at least a week of research and thought; minimally, also one round of editing. Do note, too, that this was originally written for a Malaysian audience.

Do grades matter as much as our parents say they do?

One thing we know for sure is that grades are important. Our parents reward us when we ace exams, and so we have been conditioned to think that scoring As in school are important achievements worthy of our time. And for the most part, this is true: the approval of our peers and our parents and our teachers depend largely on our grades. So grades do matter. But there's an interesting assumption to examine here: why do grades matter so much? Are they still as important now as they were 50 years ago? And perhaps - more importantly - why do so many of us accept grades as the de facto indicator of ability, without ever questioning the underlying logic of our grading systems?

I've been thinking about these questions for some time now, and I think I've come to some reasonable conclusions about them. My contention is that grades are no longer as important as they once were, and the implications from this idea are rather exciting, and scary, both at once.

Where Exams Come From
Exams and grades are relatively new things to the human experience. It wasn't until the industrial age that schools - and by extension exams - became commonplace.

The first society to implement examinations were the Chinese. Beginning with the Sui dynasty, candidates from all over China would gather to take the Imperial examinations, held once a year, every year, in order to gain entrance to the state bureaucracy.

There are two important things about exams that we must recognize began with the Chinese Imperial examinations.

The first is that the Chinese examinations served as a filter for the Imperial government. They made no pretense to be anything otherwise. And to this day exams still do act as filters: there are only so many jobs in companies and governments, and so therefore there must be a way of separating the good from the bad, the capable from the uncapable. Exams are that 'way', and they serve exactly that purpose.

The second thing may surprise you, and it is this: what the Imperialist Examinations focused on were Confucian classics. You may wonder at that, and rightly so: what does a study of the classics have to do with state management? And the truthful answer is ... well, not much. Apparently the Chinese government thought very highly of it - they considered learning the classics to be a good sign of intelligence, and if you excelled at it then you must certainly be clever enough to manage a state on your own.

Was that a valid idea? I think there's something to it - the British colonialists had a remarkably similar mentality. Do well in Latin, they say, and we'll give you a colony to rule over for us. And that worked out pretty well for them.

Where Exams Go Wrong
Those base assumptions that the Chinese Imperialists had are things that we've carried with us, even to this day. Exams are supposed to be good filters, and they are supposed to be indicative of our ability - even if 80% of what we study would never be used in our actual jobs.

But is this true?

If exams are good filters of society - how, then, do we explain the many people who do well in school, but not as well in life? And how are we to explain away the weird outliers that defy what we know of examinations - Steve Jobs, for instance, or Bill Gates, or Richard Branson? If exams are good filters, these outliers shouldn't have had happened in the first place. And if they really are good filters, everyone who scored good grades would be just as successful outside of school as they were in it.

We know that some straight-A students make for terrible employees; why is that so?

Perhaps a better question to ask is this: what exactly is it that grades measure? If exams are a filter for weeding out the good from the poor, then surely there must be something that they measure that's so valuable to future employers. If you could figure that out, perhaps you'd be able to get more mileage from your examinations.

I suspect Paul Graham has the best answer to that question: grades measure future performance. Employers are interested in future performance. They can't test everyone themselves, by putting people in the job one by one until they find the best guy, and so they rely on the next best thing - grades.

But the problem with grades is that grades aren't always a good indicator of future performance. In fact, they can sometimes be particularly bad. Grades are the results of exams; exams are a filtering system for future performance, and so therefore exams - being a system, like all systems - can be hacked.

How can exams be hacked? By its very nature, exams are particularly susceptible to people who are good at ... well, exams. In Malaysia, scoring well in exams is usually a direct function of one's ability to absorb certain facts in order to vomit them out later onto a test paper. If a student gained the skill to do exactly that - he would do well for all his exams, regardless of whether the exam was history or chemistry or math. And, yes, you'd think this would be different for subjects like math or physics, where there's thinking involved, but you'd be surprised. I remember being shocked when I first came to NUS. My discrete math teacher was a Frenchman, and the math he taught required creativity - something I had never experienced before. It was amazing and beautiful to see that coming out from a high-level math textbook, and also rather scary; scary because it wasn't particularly hackable.

Is it really a problem that grades are not good indicators of future performance? The answer to that is yes, it is - because it is no longer universally true that grades are the 'next best thing' for measuring performance. Grades were never a good indicator of future performance to start off with, and they are beginning to seem obsolete. What has changed?

The Decreasing Importance Of Grades
It would be wrong to say that companies have wised up to the inaccuracy of grades. For the large part, this change has been due to a shift in economics as opposed to a conscious, calculated decision by market leaders.

The global economy today favours smaller companies over big ones. There are many reasons for this, but the basic ones are obvious: due to globalization and cheaper logistics, it now makes more sense for companies to outsource some tasks in order to stay focused on their core competencies. This has been a relatively new phenomenon. Never before has a smaller company been able to take on a bigger company - and win. If you don't believe me, consider: in the past, railways would purchase steel refineries to manufacture the steel required for their operations; today, it outsources such operations to external - often international - companies. And this is but one example, in what is a remarkably outdated industry. The effects are far stronger and more powerful in technological companies. But, bearing these differences, what does this mean for grades?

It turns out that it is easier to measure performance in a small company, as opposed to a big one. If you do well, the company does well; if you do badly, the company does badly. This is not so for large corporations. In a large corporation, individual performance cannot be easily measured, because the organization is so large that your individual contributions do not have a significant effect on the bottom line. Graham points out this is why grades have been so important to employers for the past few decades. The harder it is to measure individual performance, the more important it becomes to predict it. And because the economy has been dominated by large corporations for the past few decades, it used to mean everything to do well in exams.

But this is fast becoming history. Grades matter less when you can directly measure what they are made to predict - which is real performance. Why bother with an indicator when you can gauge the real thing? Larger corporations are adapting in order to enjoy the benefits of being small - Google, for instance, forms small teams of engineers to develop and test new products, and presumably gauges performance based on those smaller teams.

If this seems a little incredulous to you, consider: just 50 years ago it was enough to get good grades to get a good job to retire comfortably. This no longer holds true. Today, we are not only aware of people who have succeeded despite dropping out of college, but I have friends in NUS - in Singapore, even, a very exam-oriented country - who have either skipped college altogether, or who plan to drop out of school to do their own startups. And in big companies, there are now performance reviews, where before there were none. It used to be that seniority was all that mattered; young associates accepted a lower pay because they had to pay their dues, their times had yet to come.

It seems to be uncomfortably real that grades are beginning to decrease in importance. At the very least, they no longer hold the make-it-or-break-it quality they once had. And so, if this is true, the trend begs the question: what does this mean for our students?

An Exam-Oriented Paradox
One important thing we must remember is that for certain countries, the old model seems to hold true. In Japan and Korea, for instance, if you earned good grades, you are still able to score a stable, quality job in a salaried company. And to that end both countries have very competitive exam-oriented systems, and a high density of cram school to boot.

Would this state of affairs last forever? I doubt it. It turns out that smaller companies are also more efficient companies, and globalization cuts both ways. Sooner or later, the Korean and Japanese corporations would have to outsource their operations, and therefore split into smaller, independent units, if they are to keep up with more nimble competitors. And even if they refuse to do so, current economic forces make it feasible to start small companies, particularly those built around the Internet. While Asian countries currently enjoy the luxury of the sure grade and the stable job, it is something that may not last for much longer.

Unfortunately for us, however, the Asian model of education is still optimized for this old economic model. Malaysia isn't by far the only country with an exam-oriented society. There is nothing wrong with doing well in exams; it is when exams become the focus of a society's expectations that things begin to go wrong.

The problem with exams - particularly in Asian societies - is that the examination often becomes an ends onto itself. People say that the purpose of school is to learn new things that you may put to use good later in life, but this isn't true. What you're really going to school to learn - if you're in an Asian country - is to learn how to do well in exams.

There is a simple test for this: ask yourself if studying equals learning in high school. If it isn't, and you aren't doing much learning while you study, then there is a disconnect between the two. The truth is that schools spend a lot of time teaching us how to do well in exams. 50 years ago, this made a lot of sense, but today such time is better spent perfecting ideas that would come in handy in a performance-driven world.

In Malaysian high schools, for instance, we should stop pretending that the school is there to teach us good things. They are not. The Malaysian education system, from ages 6 - 19, is designed to teach you how to do well in exams. Learning math and science and all that jazz is secondary to that one core purpose. And this is - sadly - the truth of the matter; it is why we have tuition; it is also why we have shiny ads advertising 'exam-grade' pencils and erasers on television; all of it is what it means to live in an exam-oriented society.

Creativity
There is something that must be said here on creativity. For all the noise our education system makes about creative thinking, and critical thinking, their models of teaching such forms of thinking cannot be conceptually further away from the truth.

I began this essay by asking a series of interesting questions. I examined why grades mattered so much, why grades are so important to us, and why they may be irrelevant to a reasonable assessment of success in the real world. There is one question, however, that I have neglected to answer, and that is: why do so many of us accept grades as the de facto indicator of ability, without ever questioning the underlying logic of our grading systems? I believe this to be the most important question of them all.

One reason for this may be that we have been conditioned to think, since young, that grades are the be-all and end-all of our childhood existences. But there comes a point in time in which we are old enough and wise enough to challenge our own assumptions. So now the question: why do so few of us challenge this assumption in the first place?

I suspect the main reason for this is that we are taught, since young, not to think for ourselves. We don't hear that outright, of course. No teacher actually tells us not to think for ourselves. What they do tell us, however, that has the same net effect as telling us not to think, are things like "That's very good dear, but it's not what the examiners are looking for" or "That's not a proper exam answer". I suspect that each time our teachers tell our children that, they lose the ability to think laterally, to think critically, and so gradually they don't bother to think in terms of truth at all.

End
Grades are important as measures of cognitive ability. But do grades matter as much as our parents say they do? The truthful answer is that no, they don't, not anymore. Grades don't matter as much as real-world performance does, and as it becomes easier to measure performance directly, grades will matter less and less.

But is this an excuse, then, to score bad grades? The truthful answer is that I don't know. It probably depends. It is certainly a better use of your time to go out and build things, and learn things, as opposed to spending all that time learning to score well in exams - a skill with admittedly little real-world application. But on the flip side of that coin, grades - and by extension exams - are important elements in the learning processes of our education systems. It feels like a cop-out to take such a stance after 3000 words or so of argument, but this is the truth. Just be sure that you are studying for the sake of learning, and that exams are an indicator of that learning; and not the other way around.

Saturday, March 6, 2010

Sometimes

Sometimes I wish there's an algorithm for life.

Friday, March 5, 2010

What I've Learned From Wave

24 hours after we first sat down to code in Sheares hall, I looked up and said: "I must blog about this."

"Ya, you should," Xialin said, and she looked up from her code for all of like 0.5634 seconds before looking back down at her computer screen. And I think that was the end of that conversation.

It could've been at 2 in the morning. Or 5am. Or maybe 7. Time kinda disappears when you've spent a whole night staring at monospaced fonts. I remember being hungry at around 2am. I also remember Biyan placing her head in her hands - I think I was plugging Haocong's backend into the UI then, and we both looked over at her and asked her if she was okay and she looked up and said she was thinking - but that might've been on Sunday night.

Like I said, I don't know. I can't remember. It's now all a big blur in my head.

What I do remember, however, is how much I've learnt from the Wave assignment.

What struck me the most about Wave was the sheer intensity with which Biyan and Haocong worked. I learned a lot from them. I can be very indisciplined - earlier on Saturday night I sat down to watch an animated video, putting off coding the UI, and Biyan said: "Cedric, I need to plug in my code into yours soon. So please work?" and I felt very guilty about that.

And even after I started work - and I didn't sleep like the other two - I knew that I did not match up to their level of focus. Sometime around 4 in the morning I did a count of Biyan's code and it was up to 1500 lines. Sure, there was a lot of whitespace, but my jaw dropped when I did the count, because I was quickly scrolling through the file and I figured out what she was writing and it was bloody complicated. The algorithm for check probably gave her the most trouble. And she was doing this on 2-3 hours of sleep.

Wave was also the first assignment in which I finally got to write code. As in - proper code - not the weak little descriptive languages like HTML and CSS with which I'd been playing since I was 15. Haocong said: "I need you to write this function, and you call this function from the back-end, and the parameters are this," and then he left me to do it. I really liked that. Here he was, an NOI programmer from his home country, and he trusted me to write stuff. I did it and I was quite pleased with myself after that. Learning a new programming language felt very satisfying; I'd only wished I'd done it sooner.

And there are other things, of course. Haocong spent a lot of time helping the two of us, because he finished his bit - the backend code - in about a night. I found that amazing. Haocong is probably the most understated elite programmer I've seen in 3216 so far - he doesn't say much, and he doesn't show off, but he gets things done. And then he tells you what he still needs done, instead of writing it himself, even though you probably know your code would take him all of 30 minutes to do. "When did you learn Javascript?" I asked him - and he said (without any trace of irony or inflection): "Last night."

(I think he took slightly longer to learn objective-C, but Mismatch currently has an iPhone version, and it looks great on his laptop. So I must say that, overall, it was a real learning experience working on this Wave app with him.)

Two more things.

First: our team made the mistake of waterfalling our software. I believe I've learnt my lesson - I won't do that again. Ever. Halfway through - at around 4 or 5 or 6 in the morning, I looked up from my code and said, thoughtfully: "You know what we should've done? We should've sat down together and written a whole list of interfaces before we started." And I remember Haocong sighing to that.

In the end, however, our app managed to work when we plugged things into each other the night of the deadline - but just barely. I thank God that we were writing a webapp, and not something bigger or more complex. And that became very true! Things began to be really scary when it hit 11pm and we were still nowhere near a playable chess game. Never again, I tell myself; never again.

Second: I no longer find myself insecure about my programming abilities. That is not to say that I'm great at it. I started real programming last semester, which was really late, and I found - to my surprise - how much I really enjoyed writing code. It was like finding a long lost friend. But because I was new to the school then, and new to so many things in Singapore, that I became scared and defensive when I saw all these great programmers in 1101 speaking in Java like it was their second language. They were better than me and they knew it. They didn't hesitate to talk over my head to show off how good they were with their code, and they compared lab marks with each other, and they asked me how much I got for my exams and my tutorials, sizing me up as potential competition and then then reconsidering that when they learnt I was new to this. And this made me really insecure - as silly as that may sound.

Today, however, I now know that these kids - while good - are nowhere near the level of Haocong or Biyan or Hung or Adhiraj. And that's a comforting thought. It means that I don't have to worry about those kids, for they are an order of a magnitude below than the 3216 programmers ... who are, in turn, an order of a magnitude below than the best programmers in the world. And I think that's important to know. What this means is that I needn't compare myself to them - because I know I'm new to this and I now know there's always somebody who's better than me. I have learned that what I really should be focusing on is the programming itself, because it's fun, and it's beautiful, and not the competition, or even the grades, because I shouldn't let all that taint the learning itself. And I think that is important.

Programming to me is fun and I hope to keep it that way. I'm glad I learned that from Wave. And so now: onward.

Tuesday, March 2, 2010

Holy Shit

Y-Combinator's now looking for iPad startups:
Most people think the important thing about the iPad is its form factor: that it's fundamentally a tablet computer. We think Apple has bigger ambitions. We think the iPad is meant to be a Windows killer. Or more precisely, a Windows transcender. We think Apple foresees a future in which the iPad is the default way people do what they now do with computers (and some other new things).
This took me completely by surprise. While the iPad seems more and more like a computer ordinary people would choose over the desktop/laptop, it is - to me - complete unprecedented that Y-Combinator would be so forward-thinking as to demand a startup built around the iPad. And the coolest bit about this is that they're pretty much putting their money where their mouths are. I am floored, Paul Graham. Floored.