Hacker News News
5 October 2011: Bigrss
For people who want to see more than just what's on the frontpage, we now have a variant of the rss feed that gives you 10x as many stories:
We've noticed several people have written crawlers that try to achieve the same thing by clicking repeatedly on the "more" link at the bottom of the frontpage (despite the fact that we ask crawlers in robots.txt not to follow such links). Using bigrss should be a lot faster.
Also, a traffic update: HN now gets over 120k unique ips on a weekday, and serves over 1.3 million page views.
We now finally have search for HN, thanks to Octopart, who've been working on it for a while as a test of their new search database, ThriftDB. You'll see a search box at the bottom of most pages.
ThriftDB is the software they had to write to do Octopart's electronic parts searches; no existing software was up to the task. They solved the problem in as general way as possible, producing something that will work for many different types of sites that need search.
Sorry this took so long. For a while we hoped to implement our own search, but it became apparent that would take a lot of work to do right. So we were delighted when Octopart decided to get into the search as a service business. Since this was the first search app they built outside of Octopart itself, it took quite a while. This is just the first version so it may not yet do everything users want, but it seems pretty good.
I've just released a new version that fixes much of the brokenness of the way job listings used to be handled. When I first added jobs, to save time I reused as much of the code for stories as possible. This worked well enough when HN was a small community, but it has been working less well now that HN has grown.
A lot of the newly arrived users seem not to have realized that job listings are the ads here-- that while they look like stories, they're not. The new code makes this more evident. Jobs no longer have points or submitters, and in fact the ranking algorithm for jobs is now completely different. Now instead of being ranked by points, jobs simply start at position 4 and slide down the page at 4 positions per hour. To prevent the frontpage from being overloaded with jobs, no one can post a new job when there is still one in the top half of the page.
I still have to make jobs persist longer on the jobs page. I may also have to add a job posting queue. For now it's first come first served.
As we approach HN's fourth birthday, traffic is now around 90 thousand unique visitors and 1 million page views on weekdays. (Http requests and page views are identical except for votes, of which there are about 25 thousand on a weekday.) Thanks to another round of tweaks by Rtm to FreeBSD and MzScheme, the site is as fast as it's ever been. Judging from the classic view, the stories on the frontpage are not much different from those we'd have had in the first year. And while the civility in the comment threads is sometimes strained, on the whole I'm surprised how well it has held up. So thanks everyone.
Traffic recently grew to the point where the server was getting overloaded. Most users probably noticed the site was unpleasantly slow in the middle of the (US) day. Fortunately Rtm seems to have fixed the problem:
From: rtmThanks Rtm!
Rtm just made a couple changes to Arc's internals that have made the site visibly faster. With these plus the new server, the site's faster now than it has been at any time since we launched.
We also have some new features to prevent flamewars. The most conspicuous is an exponential delay before reply links appear on deeply nested comments. We'd noticed these were rarely the most interesting comments on the site. We're hoping that instead of killing stupid arguments, it will be sufficient to apply gradually increasing drag to them.
To help users help us catch spam and junk submissions, there is now a new page listing just submissions from accounts that were less than a day old at the time: http://news.ycombinator.com/noobs. By all means help out the editors by flagging it.
Weekday traffic is now around 30,000 uniques a day.
Hacker News was launched on February 19 2007. Weekday traffic was initially around 1600 unique ips. Traffic now is around 22,000. Despite 14x growth, the character of the site has remained intact. So far this experiment seems to be working. Thanks very much, users, for making that so.
The latest version of News.YC includes several changes designed to improve or at least preserve the level of the conversation in comment threads. The most conspicuous is that some usernames now appear in (a very grayish) orange. These are the users with the highest average comment scores.
At the moment the criteria are (a) to have at least 25 comments and (b) to have an average score of at least 3.5 over your most recent 50 (or fewer) comments. Once you cross the threshold, your average has to fall below 3 for your name to revert to gray.
We're hoping that by distinguishing users with high comment scores, we'll both reward those individuals and make it easier for newly arrived users to understand the character of the site. It also turns out to be an aid in reading comment threads; comments by the high-scoring users do tend on the whole to be better, and this makes them easier to find.
To help explain the site to new users, the software now puts a link to a welcome page in the top bar and at the bottom of every comment form for the first day.
We've also doubled the karma threshold for downvoting comments (to 100), and capped the minimum score of a comment at -8.
Traffic has increased so much recently that News.YC was starting to feel noticeably slower. So we just finished a new round of optimizations as part of our ongoing quest to prove that with sufficient caching you can serve arbitrarily large numbers of requests with arbitrarily slow languages. It looks as if the site is around 25% faster at peak times. It should feel like more, though, because most of the improvement is in the generation of the pages that were slow enough to notice.
The growth has affected the character of the site slightly. The number of new accounts being created is about double what it was a month ago, and there has been a slight uptick in comments that are insulting or inane. But we're hoping that, as in past influxes, the new arrivals will with some prodding from the existing inhabitants learn the local customs.
Here's the latest traffic graph. We now get around 18k unique visitors on weekdays (including rss requests). The big bump in the spring came after News.YC was mentioned in TechCrunch. Fortunately both the character of the site and the traffic trend returned to normal afterward.
The best news here is that stories and comments haven't gotten too much worse despite all this growth. News.YC now gets around 12x the traffic it did when it was first launched, and the stories and comments don't seem too much different than they did the first week.
Growth can't keep going at this rate forever without ruining the site, though. Between those two alternatives, we prefer growth to slow down. We hope it will happen naturally—that we'll simply run out of new people the site appeals to.
The way to ensure that is to be fairly strict about keeping meretricious stuff off the frontpage. That doesn't mean keeping non-hacking stories off the frontpage, but keeping off the sort of fluff that tends to take over votable news sites because it's easy to judge (perhaps merely an image) and thus easy to upvote. If we have to crack down on anything to save News.YC from excessive growth, we'll try first to do it indirectly by cracking down on fluff posts.
A bunch of small things have been improved lately. Comment fields now last for 30 minutes. There's a "More" link on threads pages like the ones on other list pages, so you can now see the last 300 of someone's comments. There's a new "delay" field in your profile that lets you specify the delay (in minutes) between when you create a comment and when it's visible to others; this was added because many users edit comments immediately after posting them. And there's also a new "flag" link on stories and comments that lets users flag spam and trolling.
One of the problems we're working on now is the tendency for people to "pile on" and downvote comments they disapprove of to extreme negative scores. This seems to go too far sometimes. We're trying various things to fix it. One is to gradually increase the karma threshold for the appearance of the downarrow. Another is to gray down comments with negative scores to near invisibility, in the hope that this will cause downvotes to tail off.
(Another is simply to ask users: Please don't downvote comments that are only mildly mistaken or disagreeable to scores as low as -10. That kind of score should only be for spams and trolls.)
There's been one more change: user accounts are now lazily loaded, which has made server restarts more than twice as fast.
A lot of people were curious about what the TechCrunch article did to News.YC's traffic. Enough time has now passed that the oscillations have settled down, and it looks as if the result was an increase of about 20% in unique IPs per day.
There has been some dilution as a result. The stories that get voted up are not quite as good, and the average tone in comment threads is slightly less polite. But this has happened before when there were influxes new of users, and every time so far the effects have worn off as they got used to the culture of the site. Though we've never had an influx like this before, we're optimistic the same thing will happen this time.
Since polls are so common on News.YC, we just added explicit support for them. To prevent the site being swamped by polls for the next few days, we'd appreciate it if you'd only use this feature because there was something you already wanted to ask. There isn't currently any link to it, so you have to go to http://news.ycombinator.com/newpoll to make a poll, and for the time being only users with over 200 karma can make them. Once people get used to this we'll open it up to everyone.
Today is News.YC's first birthday. Over the past year, traffic has grown fairly steadily. Initially we were getting around 1600 uniques a day. Lately it's been 8-9,000. But what pleases us most is the feeling of community that has grown up here. Thanks, everyone.
Users can finally reset their passwords. There's a link on the profile page.
Users with over 250 karma will notice another field in their profile, which lets them change the color of the bar at the top. We'd long been wanting to do something special for the users who contribute most to the site, but were reluctant to do one of the usual things like display their usernames in red or put stars next to them lest new users felt their opinions counted less.
Here are the 100 most common words in submission titles, with articles, connectives, and so on removed, in order of frequency:
The latest version of News.YC is now about 50% faster. It generates the frontpage for a logged-in user in about 50 msec, vs 77 previously. For most users this will be lost in the latency, but it does at least mean the application is getting faster rather than slower.
There have been a bunch of other small improvements lately, including More links at the bottom of lists of comments.
A bunch of bugs involving edge cases in the formatting of comments got fixed during YC's annual migration. There are now no known problems. There's also finally a brief explanation of the formatting options, and a link to it next to every field supporting formatting.
Note two changes: Text has to be indented at least two spaces to be treated as code; when it was one, people kept doing it by accident. Also, an asterisk followed by a whitespace character no longer turns on italicization; people were hitting that by accident too.
Like email, social news sites can be dangerously addictive. So the latest version of Hacker News has a feature to let you limit your use of the site. There are three new fields in your profile, noprocrast, maxvisit, and minaway. (You can edit your profile by clicking on your username.) Noprocrast is turned off by default. If you turn it on by setting it to "yes," you'll only be allowed to visit the site for maxvisit minutes at a time, with gaps of minaway minutes in between. The defaults are 20 and 180, which would let you view the site for 20 minutes at a time, and then not allow you back in for 3 hours. You can override noprocrast if you want, in which case your visit clock starts over at zero.
Today is the six-month anniversary of the launch of News.YC. So far the growth rate seems to be about 5x per year. (These graphs would look more impressive if they had the usual aspect ratio.) It looks as if broadening the focus of the site has made it more engaging, because there are more page views per user, but it's still too early to say for sure.
As of today we've expanded the focus of News.YC from startup news to hacker news generally.
Recently a user asked which Startup News users had the most karma per submission. Since the server has a read-eval-print loop this is straightforward to figure out:
(bestn 20 (compare > [_ 2]) (map (fn (user) (withs (stories (keep astory (submissions user)) ns (len stories)) (list user ns (if (> ns 10) (median (map [_ 'score] stories)) 0)))) (news-users))))Here are the users with over 10 submissions who have the highest median points per submission:
(paul 43 12) (aston 12 9) (socmoth 11 9) (andrew_null 23 8) (waleedka 14 8) (nivi 12 7) (toffer 32 6) (kul 23 6) (jkush 43 5) (jkopelman 39 5) (Alex3917 34 5) (sharpshoot 85 5) (palish 31 5) (BioGeek 19 5) (brett 174 5) (dfranke 14 5) (gustaf 12 5) (pg 198 5)Congratulations to Paul Buchheit (and thanks)!
A few days ago a user methodically downmodded several other users' old comments to kill their karma. To prevent this happening in future, the downarrow now disappears off comments more than a day old. Also the karmas and story/comment scores affected have been restored to whatever they would have been without those downvotes.
We just released a new version of news.yc that fixes a bunch of small things that bothered users. Vote urls for logged in users are now static urls instead of hashes representing closures. This makes the site significantly faster, and makes hash keys expire more slowly for cases that really need them (like form submissions). We hope this will save users from seeing "expired link" after taking a long time to submit a form.
Also, there are now "More" links at the bottom of every page that displays a list of stories, not just the front page; and long urls in comments get truncated so they don't mess up the page width.
The latest version lets you delete submissions and comments.
For those who care about the details, there are now two different ways to get rid of something: to mark it as dead, which is for editors to do to spams and offtopic submissions, and to delete it, which is for submitters who change their minds.
The reason for the distinction between killing and deleting is to avoid accusations of censorship. Anyone who does want to see the stuff killed by the editors can do it by setting showdead to yes in his profile. But it seemed right to offer submitters a more thorough sort of deletion for their own stuff.
Sorry this took so long. Let us know if you notice any breakage.
The latest version of News.YC has a new feature: a jobs page where you can see jobs posted by startups we've funded.
The next best thing to starting a startup is to go work for one. Being one of the first hires at a new startup is very much like being a founder, both in what's expected of you and in how well you can do if the company succeeds.
Some of the startups we've seed funded are now established companies. Others are still just a couple guys in an apartment. What all the companies on news.ycombinator/jobs have in common is that they impressed us enough to bet our own money on them. So if you want to join a startup, there is probably no more concentrated source of good leads.
We just launched a new version that is 2-3x faster, thanks to Robert
Morris, who rewrote some of the innards of Arc not to cons so much.