Planning graphs have been shown to be a rich source of heuristic information for many kinds of planners. In many cases, planners must compute a planning graph for each element of a set of states, and the naive technique enumerates the graphs individually. This is equivalent to solving an all-pairs shortest path problem by iterating a single-source algorithm over each source. We introduce a structure, the state agnostic planning graph, that directly solves the all-pairs problem for the relaxation introduced by planning graphs. The technique can also be characterized as exploiting the overlap present in sets of planning graphs. For the purpose of exposition, we first present the technique in deterministic planning. A more prominent application of this technique is in belief state space planning, where an optimization to exploit state overlap between belief states results in drastically improved theoretical complexity. We describe another extension in probabilistic planning that uses common action outcome uncertainty to further improve theoretical complexity. Our experimental evaluation (using many existing International Planning Competition problems) quantifies each of these performance boosts, and demonstrates that heuristic belief state space progression planning using our technique is competitive with the state of the art.