The pin layout is identical, unfortunatly the physical construction differs completly.
I used OpenCV with Python and Java only, so I never had to deal with porting code between different architectures or operating systems.
About the compiling and "they are all x86", there are two issues:
The Quark SoC is - exaggerated - some kind of milkshake of the i486 and i586 architecture. Some configure scripts may fail to prepare the build as needed, even if the build chain for cross compiling is set up correctly. They may - I guess - create an i586-only binary instead of sticking to i486 functionality where necessary. An example is ffmpeg. It is part of the (LSB) Linux image, it was cross compiled on a host computer without problems, but the binary fails on the Galileo because of "Illegal instructions" (means usually "unknown/misused cpu op codes"). I got a partly working ffmpeg after compiling it on the Galileo itself.
Another problem, and that is the main issue with existing binaries not working - the Linux images by Intel uses the uglibc, while most (desktop) linuxes use eglibc as base library. The later can be solved by using the LSB Linux image.
Edit: Intel has a word filter in the forum...
