Using endfloat with appendices

I am not completely sure how to articulate this question, but:

I am working on a manuscript with figures that are discussed in the main text and should be included with the main text, as well as figures that are briefly referred to in the main text but are relegated to an appendix.

The beauty of endfloat is that as long as I place a float close to the first place it is referred to in the text, it will appear in the correct order and the references in the text will be generated in the correct order (the first figure included will be "Figure 1", the second "Figure 2", etc.).

I would like to extend this to figures that are referred to in the main text but appear only in the appendix, and in the order it is referred to in the text ("Figure A1", etc.).

Here is an MWE, where I have to place the appendix figure into the appendix, rather than let endfloat do the work for me:

\documentclass{article}

\usepackage[noheads,nolists,tablesfirst,nomarkers]{endfloat} 
\usepackage{appendix}
\begin{document}



\section{Main Text}

Our first figure is Figure \ref{fig:first}.

\begin{figure}
    \centering
    \caption{First figure}
    \label{fig:first}
    FIRST FIGURE
\end{figure}

I would like for the next figure to go into the appendix and be called Figure A1. However, to do that, I have to manually put it after I call the appendix, like I've done here: Figure \ref{fig:appendix1}.


Our second figure in the main text is Figure \ref{fig:second}.

\begin{figure}
    \centering
    \caption{Second figure}
    \label{fig:second}
    SECOND FIGURE
\end{figure}

\processdelayedfloats


\appendix 

\section{My Appendix}

\setcounter{figure}{0}
\renewcommand{\thefigure}{A\arabic{figure}}

\begin{figure}
    \centering
    \caption{First appendix figure}
    \label{fig:appendix1}
    FIRST APPENDIX FIGURE
\end{figure}


\end{document}

And here is something like a minimal non-working example:

\documentclass{article}

\usepackage[noheads,nolists,tablesfirst,nomarkers]{endfloat} 
\usepackage{appendix}
\begin{document}



\section{Main Text}

Our first figure is Figure \ref{fig:first}.

\begin{figure}
    \centering
    \caption{First figure}
    \label{fig:first}
    FIRST FIGURE
\end{figure}

I would like for the next figure to go into the appendix and be called Figure A1. However, enfloat does not know that and just calls it Figure 2: Figure \ref{fig:appendix1}. 

\begin{figure}
    \centering
    \caption{First appendix figure}
    \label{fig:appendix1}
    FIRST APPENDIX FIGURE
\end{figure}

How can I tell endfloat that the above should go into the appendix? 

And now our second figure in the main text will be called Figure 3, like this: Figure \ref{fig:second}.

\begin{figure}
    \centering
    \caption{Second figure}
    \label{fig:second}
    SECOND FIGURE
\end{figure}

\processdelayedfloats


\appendix 

\section{My Appendix}

\setcounter{figure}{0}
\renewcommand{\thefigure}{A\arabic{figure}}



\end{document}