Counting crop seedlings is a time-demanding activity involved in diverse agricultural practices like plant cultivating, experimental trials, plant breeding procedures, and weed control. Unmanned Aerial Vehicles (UAVs) carrying RGB cameras are novel tools for automatic field mapping, and the analysis of UAV images by deep learning methods can provide relevant agronomic information. UAV-based camera systems and a deep learning image analysis pipeline are implemented for a fully automated plant counting in sugar beet, maize, and strawberry fields in the present study. Five locations were monitored at different growth stages, and the crop number per plot was automatically predicted by using a fully convolutional network (FCN) pipeline. Our FCN-based approach is a single model for jointly determining both the exact stem location of crop and weed plants and a pixel-wise plant classification considering crop, weed, and soil. To determinate the approach performance, predicted crop counting was compared to visually assessed ground truth data. Results show that UAV-based counting of sugar-beet plants delivers forecast errors lower than 4.6%, and the main factors for performance are related to the intra-row distance and the growth stage. The pipeline’s extension to other crops is possible; the errors of the predictions are lower than 4% under practical field conditions for maize and strawberry fields. This work highlight the feasibility of automatic crop counting, which can reduce manual effort to the farmers.