This is just part of what being a language user involves.
Even if they are not always part of linguistic meaning, primary intensions are nevertheless a sort of truth-conditional content.
On the other hand, if we stipulate that semantic content is linguistic meaning in the sense above, or that semantic content is always associated with expression types and not tokens, then primary intensions are not in general part of semantic content (though they may be part of semantic content for some expressions, such as some indexicals and qualitative expressions).
For example, one might hold that the primary intension of a mental state is determined in part by its internal functional role, and in part by associated phenomenal states (where the latter may be especially relevant for phenomenal and perceptual concepts).
So an expression's primary intension is not part of its linguistic meaning, where this is understood as meaning that is associated with an expression type simply by virtue of the conventions of a language.
Response: This point is correct: primary intensions are not always part of linguistic meaning.
The inferences are justified in part by background empirical knowledge of the world (Block and Stalnaker) or by "peeking" at our own judgments (Yablo).
While the latter employs all the richest colours of his art, and gives his figures the most graceful and engaging airs; he must still carry his attention to the inward structure of the human body, the position of the muscles, the fabric of the bones, and the use and figure of every part or organ.
Here indeed lies the justest and most plausible objection against a considerable part of metaphysics, that they are not properly a science; but arise either from the fruitless efforts of human vanity, which would penetrate into subjects utterly inaccessible to the understanding, or from the craft of popular superstitions, which, being unable to defend themselves on fair ground, raise these intangling brambles to cover and protect their weakness.
Besides this advantage of rejecting, after deliberate enquiry, the most uncertain and disagreeable part of learning, there are many positive advantages, which result from an accurate scrutiny into the powers and faculties of human nature.