Welcome to Information Processing Lab Home Page. 

Main research thrust of our current work is in the multimedia signal processing, multimedia networking and machine learning areas. A specific focus is on the large scale smart camera networks analytics and networking. Details about various projects and research can be seen using the links on the left.

The research group works under the guidance of Prof. Jenq-Neng Hwang and Prof. Ming-Ting Sun.

Information Processing Lab

Copyright © 2015, Information Processing Laboratory, University of Washington

Our research paper, 'Camera Self-Calibration from Tracking of Moving Persons', has won two Finalist Best Student Paper Awards in Track 3 (Image, Speech, Signal and Video Processing) of the 23rd International Conference on Pattern Recognition (ICPR) sponsored by INTEL and IBM.

Our research group has achieved the top performance on the benchmark dataset, NLPR_MCT, for multi-camera object tracking (http://mct.idealtest.org/Result.html). The corresponding results are published in IEEE Transactions on Circuits and Systems for Video Technology and 2017 IEEE International Conference on Image Processing.


Our research group is the Winner of Track 2 (AI City Applications Track) at the 2017 IEEE Smart World NVIDIA AI City Challenge. 29 teams taking on this challenge hail from academic labs around the globe—Brazil, China, Greece, India, Italy, Japan, Turkey and the United States. 18 teams from 15 universities, including UC Berkeley, UIUC, SJSU, SUNY, etc., went on to the final demonstration in Fremont, California—USA. Our team is selected as the winning team for the value and innovation of our proposed approach, along with the success of our demonstration. The team members include Zheng (Thomas) Tang, Gaoang Wang, Tao Liu, Young-Gun Lee, Adwin Jahn, Xu Liu, Dr. Xiaodong He from Microsoft Research and Prof Jenq-Neng Hwang. Here is the link to the report at the official blog of NVIDIA: https://blogs.nvidia.com/blog/2017/08/09/ai-city-challenge/

Acknowledgment: The 3D CMK vehicle tracking framework is developed based on the implementation by the honorable graduates from the Information Processing Lab, Dr. Kuan-Hui Lee and Dr. Chun-Te Chu. This work also cannot be done without the assistance of our undergraduate researchers, including Lingli (Fiona) Zeng, Aotian Zheng, Yan Kuo, Kevin Nguyen, Jingwen Sun and Chien-Jen (David) Hwang.

The team from the Information Processing Lab representing the University of Washington is the winner of Track 1 (Traffic Flow Analysis) and the winner of Track 3 (Multi-camera Vehicle Detection and Reidentification ) at the AI City Challenge Workshop at CVPR 2018. There have been 22 teams from all over the world submitting there results. Our team achieves rank #1 in both of the leaderboards of Track 1 and Track 3. The team members include Zheng (Thomas) Tang, Gaoang Wang, Hao Xiao, Aotian Zheng and Prof Jenq-Neng Hwang. The published paper can be accessed here. The source code can be downloaded here.

Our story has been reported by the Department of Electrical Engineering here. The links to view our demo videos are as follows: demo of vehicle tracking and speed estimation and demo of vehicle re-identification.

Acknowledgment: We would like to thank many people who helped in the improvement of the performance of the proposed system: Chen Bai, Ge Bao, Jiarui Cai, Bill Cheung-Daihe, Tianhang Gao, Yijin Lee, Ching Lu, Shucong Ou, Ningyang Peng, Mingxin Ren, Jingwen Sun, Yi-Ting Tsai, Yun Wu, Chumei Yang, Xiao Tan, Hao Yang, and Jiacheng Zhu.

Professor Hwang’s interview by The Wall Street Journal


The Wall Street Journal, April 1st, Business/Journal Reports: Technology

How Many Fish Are There in the Sea? AI Can Find the Answer

Cameras, satellite imagery and computers are helping to better understand the health of global fish stocks, as well as prevent illegal fishing


Chris Kornelis

April 1, 2019 5:46 p.m. ET

SEATTLE—Sitting behind his desk at the University of Washington’s electrical and computer engineering department, Jenq-Neng Hwang watches a video of a fish flopping as it’s pulled out of the water on a commercial fishing boat.

It’s probably a halibut. He’s not exactly sure. His job isn’t to know the species of every fish caught on such boats. It’s to create machines that know—computers that use machine learning to instantly distinguish between hundreds of species, count and measure the length of each fish and record where it was hauled in—all while the fish are very much not holding still.

“If all this information can be estimated in real time on thousands of fishing boats, then we are doing real-time monitoring of the whole ocean,” says Prof. Hwang, whose work is part of the Electronic Monitoring Innovation Project through the National Oceanic and Atmospheric Administration’s Alaska Fisheries Science Center.

How Many Fish Are There in the Sea? AI Can Find the Answer

Scientists, engineers and fishermen and women around the world are deploying machine learning and computer vision to better understand the health of global fish stocks and attempt to preserve their future productivity. Cameras, satellite imagery and computers guided by artificial intelligence all have roles not just in identifying and measuring daily catches, but in tracking possible illegal fishing as well.

Counting fish

Peter Mous, director of the Fisheries Conservation Program at the Nature Conservancy in Indonesia, works with local fishermen to try to understand whether, or to what degree, fish stocks in the region are overexploited. For the moment, most of this work is still done manually. Technicians for the conservancy in Indonesia examine digital photographs of fish, mostly snappers, taken by the crews of about 300 boats, or 3% of the roughly 10,000 fishing boats in Indonesia.

Snappers are some of the most sought-after fish in these waters. If most of the snappers in the photos are fully grown, Mr. Mous says, that suggests the snapper stock is in good condition. If the boats are catching small fish, he says, it generally means the stock is overfished. In the enormous area that Mr. Mous is studying, covering three time zones, he says there’s a mix of stocks. Some are healthy, but others consist primarily of juvenile fish—suggesting overfishing in those areas.

Within five years, Mr. Mous wants to be working with 5% of the fishing fleet. But it will be difficult and not very cost-efficient, he says, to keep examining the increasing load of photos manually. In addition to the volume, even professionals have trouble keeping track of all the different species present in Indonesian waters, which Mr. Mous calls, “in terms of marine life, the global center of biodiversity.”

To be able to step up its monitoring capacity, the conservancy has enlisted a Swedish company, Refind Technologies, to help it develop an automated system, called FishFace, that uses machine learning and computer vision to identify and measure fish and count catches.

The programmers of FishFace, which is currently deployed on one boat, don’t tell it what characteristics to look for to identify a fish. Instead they show it thousands of pictures of fish whose species are identified, then let the program figure out what characteristics it will use to differentiate each species.

The goal, Mr. Mous says, is to be able to present data to Indonesia’s Ministry of Maritime Affairs and Fisheries in support of setting some limits on catch sizes, limiting numbers of boats or perhaps shortening fishing seasons. In Indonesia, he says, aside from a few protected areas, there are no limits on what fishing boats can catch.

In time, Refind co-founder Johanna Reimers says, the data collected by projects like FishFace could be used to certify that fish bought at a supermarket in, say, Montana, was legally caught halfway around the world.

Counting boats

In addition to counting fish, machine learning can help monitor boats. Global Fishing Watch is an international nonprofit that seeks to monitor, expose and prevent illegal commercial fishing and raise awareness of unmanaged overfishing. Offshore from countries like Namibia, for example, with some 900 miles of coastline and nowhere near the monitoring resources of richer countries, it’s easy for unlicensed ships to come in, scoop up fish and disappear.

Using machine-learning programs that analyze data from sources including radar and GPS, the group creates a map that shows all vessels over 49 feet in length that it can find. It also shares information privately with governments in places like Peru, Indonesia and Panama to help them use their limited resources to go after possible illegal fishing.

The nonprofit’s computers can analyze the movements of boats and the equipment they use to determine what they’re fishing for, and whether they’re licensed to do so.

“You can make a pretty good guess about what kind of species they’re targeting based on what species is likely to be there and can be caught with that kind of gear,” says Paul Woods, chief technology officer.

Mr. Woods says that within 10 years, as his group gets more data—including from governments about their countries’ fisheries—Global Fishing Watch hopes to be able to detect “everything that’s extracting anything from the ocean and make that generally transparent and publicly visible.”

Increasing transparency, he says, might entice action by governments that aren’t inclined to admit to a problem with illegal fishing.

Global Warming

Improving the counting and examining of commercially caught fish in real time will be a big help in tracking the health of global fish stocks. That data may also yield insights possibly resulting from the effects of global warming. As the climate changes and fish disappear from one area and reappear in another, for example, data gathered by monitoring could be used to help direct fishermen to areas where they are likely to catch more fish.

Farron Wallace, a fisheries biologist in the North Pacific Observer Program at NOAA’s Alaska Fisheries Science Center, is leading AFSC’s Electronic Monitoring Innovation Project, working with Prof. Hwang at the University of Washington, to use computer vision and machine learning to eventually keep track of stocks in real time.

The planet is getting hotter every year, Mr. Wallace says. “We need to have much better information…to really do a good job of trying to understand how climate change is going to affect those resources out there.”

Mr. Wallace and Mr. Mous agree that global warming isn’t expected to negatively affect populations of all species of fish.

“The whole global warming is very much a wild card,” says Mr. Mous. “You know that things will change in fisheries, but you don’t know exactly how. For that reason, it is important also to keep a finger on the pulse.”

Mr. Kornelis is a writer in Seattle. He can be reached at reports@wsj.com.