\documentclass[11pt]{article}

\usepackage{graphicx}
\usepackage{fullpage} % This is to change the margin
\usepackage{subfigure}
\usepackage{textcomp}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{color}
\usepackage{epsfig}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\newcommand{\response}[1]{\textcolor{blue}{\em \noindent #1} }
\newcommand{\added}[1]{\textcolor{blue}{#1}}
\newcommand{\changed}[1]{\textcolor{red}{#1}}
\newcommand{\removed}[1]{\textcolor{gray}{#1}}

\DeclareMathOperator{\BigO}{O}
\newcommand{\eg}{e.g.\!}
\newcommand{\Eg}{E.g.\!}
\newcommand{\etal}{et al.\!}
\newcommand{\ie}{i.e.\!}

\begin{document}

\noindent
Dear editors \& reviewers,
\vspace{0.5cm}

\noindent
{\bf RE: Decision on your manuscript VISI-D-12-02746, ''Active Rare Class Discovery and Classification using Dirichlet Processes'' by Tom S. F. Haines and Tao Xiang.}
\vspace{0.5cm}

We would like to thank the reviewers for their additional constructive feedback and suggestions.
The manuscript has been revised accordingly.
Details of our changes and our responses (in italics) are elaborated below.
We hope that our revised manuscript is acceptable for publication.



\vspace{0.5cm}
\noindent {\bf Reviewer 1:}

\begin{enumerate}

\item (1a) The authors clearly state in their response that "we are the first and only authors to use [Dirichlet Processes] for the purpose of active learning." This is not true, see [1] below (I also found several papers very close to the subject, but not quite close enough to warrant includion here).

[1] Vlachos, Andreas, Zoubin Ghahramani, and Ted Briscoe. "Active learning for constrained Dirichlet process mixture models." Proceedings of the 2010 Workshop on GEometrical Models of Natural Language Semantics. Association for Computational Linguistics, 2010.

\response{The paper you have suggested ([1], above) uses Dirichlet processes for the purpose of clustering, and feeds the sampled probabilities into an active learning method, specifically the standard entropy approach. This is not the same thing as using a Dirichlet process for active learning directly, and hence it does not undermine our statement. Note that we could swap the entropy approach for our own, and continue to use the weakly-supervised DPMM for the clustering without alteration. This would of course be pointless, as it is an example of active classification alone, whilst we are concerned with a different problem, of active discovery and classification with rare classes. It remains the case that only one previous approach (Hospedales et al. 2011) attempts to solve the presented problem, and does so in an entirely different (and inferior) manor.}


\item (1b) Dirichlet Processes are commonly used to estimate (eg) the number of components in a mixture model. The active learning element in the paper amounts to computing the probability that an instance is wrongly classified. Since this is given by the DP under two simplifying assumptions made by the authors, they are correct to write "It is then a simple matter to calculate the probability of incorrectly classifying an instance, …".
   Even if we disregard the existence of [1], the degree of novelty is too small, in my view, for IJCV. Selecting the worst-case instance as the next candidate does make sense; in fact it a heuristic I have used myself. However, I regard that heuristic as being too obvious for a strong claim of novelty to be made.

\response{This is simply a rewording of your complaint from the first review - the argument against it remains the same. We are only the second group to consider the active discovery and classification problem, and have presented a novel and elegant solution. The fact it is simple would usually be considered an advantage, as it makes it easy to implement. The complexity of an approach is irrelevant when judging novelty. For a more in depth rebuttal please refer to the previous one.}


\item Previously I was concerned that the paper does not match the journal. The authors responded by writing "[the] ability to solve more general (machine learning) problems should preclude [our method] from continuing to be considered as a computer vision paper."
Of course, I agree with them - but they miss my points.

My points are: \\
(a) that ML papers that claim to be general are best reviewed by the general ML community, \\
(b) ML papers in CV should be applied to CV problems.

I made one suggestion before regarding segmentation, but active visual-object class discovery is another. As it is, the authors continue to regard the data as a passive entity in the sense that it is already given by some external process. I am not suggesting the authors invent new CV, only that they apply their active learning method to a CV problem in a more intimate way.

\response{We do agree that submitting to a ML journal would make sense, and were undecided about whether to go for a CV or ML journal prior to being invited to submit to IJCV. However, we still feel we have made a significant contribution to the CV community, as our approach is highly applicable in this area. Regarding your suggested experiments, they are not that different from what we are already presenting. The \emph{segment} problem already involves querying for the class labels of regions of an image. Whilst the relatively simple interaction implied by our experiment could certainly be refined (It would be silly to ask a human to label just one segment of an image before moving on.) this would be orthogonal to the presented approach, and it would only confuse the paper whilst contributing no further knowledge or insight about $P(\mbox{wrong})$. We could run on a data set similar to the segmentation problem with many more classes and treat it as a class discovery problem, but again it would not contribute anything we have not already demonstrated.}


\end{enumerate}



\vspace{0.5cm}
\noindent {\bf Reviewer 3:}

\begin{enumerate}

\item - P3, L12-18:  Tong and Koller (2000) did not introduce the concept of "version space" (it was introduced earlier by T. Mitchell, Artiﬁcial Intelligence, 1982).  I suggest rewriting that sentence to something like, "Tong and Koller (2000) introduce a margin-based active learning method for SVMs, which attempts to minimize the version space of the SVM at each iteration..."

\response{Done, with the correct citation used for version spaces.}


\item - P7, L31-32:  While this may be true, there is no empirical evidence in the paper to support this vague claim.  One suggestion to revise would be to point out that typically an estimate of confidence can still be obtained with non-Bayesian classifiers.

\response{Done.}


\item - P7, L51-55:  The section on "Cost-benefit analysis" is unclear.  (Should "maximized" be "minimized"?)  Please elaborate.

\response{Have revised this section, and corrected it to minimised. As it is an application of cost benefit analysis from economics a suitable reference has been included and an example scenario presented.}


\item - P11, footnote 10:  It would be better to omit this; it does not add anything to the paper.

\response{Agree - have removed.}


\item Minor comments: \\
- P2, L15: its $->$ it is \\ % done
- P6, L19: italicized P(data) (other places it's not italicized) \\ % done
- P8, L23: 2(a) $->$ Figure 2(a) % done

\response{All of the above changes have been made.}


\end{enumerate}

\end{document}
