If your problem is in the form of separating circles from rectangles. That can be quite tricky, but can be done in a number of ways. A good start point is using something like Fred suggested to separate each shape from the other shapes, so you have one image per shape. That will greatly simplify your analysis.
After that...
rectangles by doing a hough transform (not built into IM) and looking for 4 lines,
circles by doing a circular hough transform.
Morphology can also help in this regard, specifically for more general analysis of 'random' shapes. For example it can grade shapes by size, lengths, how far the 'center' is from nearest edge using a
Distance Gradient. There may be multiple centers (peaks). Or in the case of a rectangle a line of centers of equal distance from an edge - regardless of rotation. Similarly for glyphs (letters or symbols) that was draw using a pen of constant thickness you will get lots of 'centers' that have very similar distances from edges, and a larger center around where lines join.
Once you have a center you can reverse the distance gradient to find how far the furthest edge is from that center, as the crow flys, verse how far it is via a internal path through the shape, using
Constrained Distance. If the two are very different, then you not only have 'arms' or 'psuedopods' sticking out of the shape, those arms curve around or even curve back towards the center.
Separating the random shapes from others is harder. One way is to generate a morphological skeletion, and then analysing those lines to determine ... number of arms, whether the shape has 'holes' in it and how many. etc..
http://www.imagemagick.org/Usage/morpho ... g_skeleton
Fractal analysis may also help but I am not certain how to use it, but I mention it as another line of study for shape analysis.
