We provide annotated fish imagery data for use in deep learning models (e.g., convolutional neural networks) for individual and species recognition. For individual recognition models, the dataset consists of annotated .json files of individual brook trout imagery collected at the Eastern Ecological Science Center's Experimental Stream Laboratory. For species recognition models, the dataset consists of annotated .json files for 7 freshwater fish species: lake trout, largemouth bass, smallmouth bass, brook trout, rainbow trout, walleye, and northern pike. Species imagery was compiled from Anglers Atlas and modified to remove human faces for privacy protection. We used open-source VGG image annotation software developed by Oxford University: https://www.robots.ox.ac.uk/~vgg/software/via/via-1.0.6.html.