So, what defines a literary speculative fiction (which would cover both literary fantasy and literary science fiction)? Is that an achievable goal? I think that it is.
In goal-oriented speculative fiction, the adventure has a clearly announced endpoint and identifiable milestones. Milestones may appear or disappear (as plot twists) but the end tends to remain the same. Vast amounts of fantasy and SF fall into this fold. Star Wars, the Secret of NIMH, Watership Down, Jurrasic Park, Around the World in 80 Days, and The Avengers all fall into this rubrik.
Goal-orientation is a huge field by itself. Murder mysteries, horror, and thrillers all fall into the goal-oriented field of fiction. It should be no surprise that speculative fiction can morph easily into all of these.
In literary speculative fiction, the structure is a life arc of a character. The character's experience has a beginning, a middle, and an end, but the apparent plot itself does not follow a firm structure, often feeling episodic. It lacks the milestone and fixed goal structure of a quest.
A grant example of literary fantasy is the Earthsea series by Ursula LeGuin. Although we see many parts of our characters, there's no goal to tell us when we're done. Their ultimate journeys are not outside, but inside. Every character passes through the markers of adulthood, from the formal ones that say that they are adults, to the spiritual ones that mark the completion of that transition.
The great difficulty of pinning down literary fantasy further is that the topics are experiential. They are about the inner character, a topic which is as wide as the sea is deep. Keeping the local action interesting while traversing across the character development arc is considerably more difficult than plotting a goal-oriented book.
While not mutually exclusive, as each sort of fiction include both goals and character arcs, the difference is the focus of the narrative and what feels like a satisfactory conclusion to the work.