Yes, a bit of that! If the fishing vessel is emitting AIS data you will already have the code from the AIS transmitter as well as the IMO number (like a social security number but for ships), and by using these numbers you can see whether it is a fishing vessel or not. These can ofcouse be spoofed in the case of illegal fishing vessels, but I think the likelihood of this happening in the terretorial waters of a country with an established coastal authority is slim.

In that case it is more about identifying the actual fishing activity than the fishing boat itself. Each mode of fishing (which requires different types of gear and types of ships) will leave a distinct signature. For example bottom-trawling will require a certain (low) speed (here a big net is dragged behind the vessel), purse seines will make somewhat circular patterns (they lay out a net in a circle around a school of fish). So often you want to identify these fishing activities to, for example link them up with a later delivery of catch, but also to investigate and estimate things like fuel usage.

Even if you don't have the IMO number of the ship you can look for these patterns, so it can be redused to whether the ship is in proximity of a fishing ground, if they display the movement patterns that is the signature of fishing activity, and later if they meet up with other vessels or go in to land to deliver their catch. A recreational boat (given that it is large enough to be suspected to be a fishing vessel just by where it is and how large it is) would perhaps have a constant high velocity (or complete stand still) for instance.

A simplified example: a person is jogging on the road that passes your apple trees you would not suspect them of stealing apples, if they are slowly walking - maybe (but if they stay no the road - probably not), if they are jogging through your garden of apple trees its certainly suspicious, but if they are slowly walking through your garden I would send a coastal guard ship ;)

A less simplified one would be to try to match up fishing related activity with how much fish a vessel is delivering (is it delivering 20 tonnes but have spent two full weeks with fishing activity?). Then maybe see if they have the right permissions to do that kind of fisheries in that zone at that time of the year etc.