Skip to content

Conversation

@robwhess
Copy link

Note: sorry to not have used the typical pull-request-first-then-develop approach. We had already developed this code and thought it might be a useful contribution back to the main repo.

This pull request generalizes the accuracy layer to compute the top-k accuracy instead of the top-1 accuracy. Specifically, a testing example can now be counted as a correct classification if its true label falls within the top k predicted labels. This will be useful, for example, in comparing results to ImageNet benchmarks, which are typically reported as top-5 accuracy/error.

To use, add to your network proto an ACCURACY layer with an accuracy_param (new) with a top_k element. For example:

...
layers {
  name: "accuracy"
  type: ACCURACY
  bottom: "prob"
  bottom: "label"
  top: "accuracy"
  accuracy_param {
    top_k: 5;
  }
}

The default value for top_k is 1, which replicates the current behavior under dev (i.e. this update is backwards-compatible with previous versions).

@shelhamer
Copy link
Member

No worries about the protocol -- this is a welcome contribution. Thanks @robwhess!

Note that this needs to be coordinated with #522 which splits the accuracy from loss in AccuracyLayer. We'll merge these PRs in order. You might need to do a slight adjustment once #522 is in but it should be easy (if there are conflicts at all).

@sguada
Copy link
Contributor

sguada commented Jun 26, 2014

Now that #522 is merged, could you adapt the top_k code? and rebase against dev again?

If you don't want to do it I can do it myself.

@robwhess
Copy link
Author

@sguada, I rebased on dev, though something seems to have gone wrong, since each commit here is now replicated twice in the log. Anyway, the status of the code is correct. Let me know if there's an easy way to fix the log.

@shelhamer
Copy link
Member

Oh, it looks like you did a merge instead of a rebase or something to that effect. Try this

# just to be sure you're up-to-date
git checkout dev
git pull
# file this weird branch as "broken" for reference
git branch -m dev-top-k-accuracy broken
# branch off the original end of your feature branch, then rebase to dev
git checkout -b dev-top-k-accuracy 5fc2682
git rebase dev

# ...fix any conflicts by editing, git add, and git rebase --continue

# overwite your current PR with the correct rebase
git push -f flickr dev-top-k-accuracy

and if that doesn't work give a shout.

EDIT: swapped the order of the treeish and new branch name for correctness.

@robwhess
Copy link
Author

Thanks @shelhamer. It would have taken some work to figure that all out on my own. The rebase should be done correctly now and the PR ready to go.

shelhamer added a commit that referenced this pull request Jun 27, 2014
Measure accuracy as top-k (default to top-1)
@shelhamer shelhamer merged commit a7692e8 into BVLC:dev Jun 27, 2014
@shelhamer
Copy link
Member

Thanks!

@robwhess robwhess deleted the dev-top-k-accuracy branch June 27, 2014 21:00
@shuokay shuokay mentioned this pull request Jul 3, 2014
@shelhamer shelhamer mentioned this pull request Aug 7, 2014
mitmul pushed a commit to mitmul/caffe that referenced this pull request Sep 30, 2014
Measure accuracy as top-k (default to top-1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants