This project developed and deployed our prototype cloud-based AI-assisted electronic monitoring (EM) review system. Deployed in a dedicated Microsoft Azure environment, the system features cloud-hosted secure video storage; an Application Programming Interface (API) supporting AI analysis of video for EM vendors with their own review platforms; and a web-based application supporting AI-assisted video review in fisheries without current EM programming. The web-based application developed in this project evolved into our CatchVision software. The project aim was to improve efficiency in EM review and it delivered almost a 50% reduction in reviewer time.
Our initial API prototype and associated infrastructure contained the basic components required to support the successful upload, processing, and download of results from a single video. AI processing was automatically triggered as files arrived. Results were available in a JSON output format. Functionality was then expanded to handle a batch of video such as a day of video, or a full fishing trip, and a CSV results format.
AI-assisted review of EM footage that replicates target catch review by human reviewers involves 3 tasks. First the AI must detect fish present in the footage, it must track the fish through the video to ensure accurate counting, and it should classify the fish species. For this project we focused on the first two tasks of object detection and tracking, leaving classification to a human reviewer. A/B testing was conducted on 6 trips of footage. This testing involved trained reviewers conducting a review of the footage following standard procedures for their usual review. An additional AI-assisted review was then performed to compare review efficiency. Two different reviewers conducted each type of review on each trip and recorded their outputs for each trip and time spent on the review.
The Ai.Fish API Workflow
The project achieved a mean 48% reduction in review time. This equated to a 46% reduction in review given AI processing costs. AI-assisted review had a mean error of 2.7% while human reviewers differed by up to 1.3%.
The project took place in the Halibut and Sablefish fishery. 16 trips of training data were collected from the fishery. Our in-house annotation team built a library of over 500,000 annotations from this footage. Algorithms were developed to detect humans and fish. A tracking approach was implemented to support accurate counting. An AI pipeline was implemented to support cloud-based processing in a parallelized manner. We achieved an average precision of 90% in model testing. Key challenges for the algorithm included the unique shape of halibut as a flatfish and the occlusion of fish during fishing activity.
.png)
.png)
Artificial intelligence has a significant potential to improve the cost efficiency of electronic monitoring review programs including opportunities to ensure the value of human review is maximized.
.png)