After spending years stealing market share from bulge-bracket peers on Wall Street, many boutique investment banks are stalling as a lengthy merger boom sputters.