If the President neglects his duties, despite the fact that the authority is conferred upon him, he lacks the power associated with the assignment. His keys become thereby wasted or lost.
Keys, however, are not limited to the church giving an assignment. When the Lord gives an assignment, commission or commandment to a person by His own voice, then the Lord similarly gives to the person the keys to accomplish the assignment, commission or commandment. By acting consistent with the duty devolving upon him, the man receives not only the larger assignment, but also the inspiration to accomplish the smaller or more detailed activities related to the assignment given to him.
An example from Nephi illustrates the point. Nephi was commanded to build a ship. (1 Ne. 17: 8.) Nephi needed direction and instruction to accomplish the task given to him. Since he possessed the keys to accomplish the work, the direction was forthcoming from the Lord as it was needed and as Nephi inquired to obtain it. (1 Ne. 17: 9-10.) In the process of asking and receiving direction as he fulfilled the assignment, Nephi learned other, greater things as well. (1 Ne. 18: 2-3.)
Nephi saw in the assignment (keys) he had been given a direct relationship between fulfilling the assignment to build a ship and Moses' commission (keys) to deliver Israel from bondage. He used Moses as an example to his brothers to justify how the Lord could assign someone as untrained as Nephi to build a ship. (1 Ne. 17: 23-43.) It was an appropriate example. It illustrates how once the Lord gives an assignment to a man, the Lord entrusts the keys and provides the inspiration to accomplish the assignment.
Similarly, all the prophets who have been sent to warn Israel in any generation have been given the keys from God to accomplish their assignment. Even among people who no longer held such authority, the Lord would directly ordain those He commissioned during the Old Testament times. (TPJS p. 181.)
When the church builds a temple and calls a temple president the one called to preside over the temple is the only one who can organize and run the temple. He has the keys and should be respected. Anyone who has an assignment or keys conferred upon them, by the church or by the Lord, has an assignment that should be respected.
Nephi's brothers and the royal court of King Noah all learned that it simply wasn't possible to terminate the mission of someone holding keys before they finished their assignment. (For Nephi, see 1 Ne. 17: 48-55. For Abinadi see Mosiah 13: 2-5.) Of course, once the assignment given the man has been completed, they are as vulnerable to destruction at the hands of enemies as anyone else. (Mosiah 17: 20; D&C 135: 4.)
When someone receives an assignment, and fulfills it with honor, they hold the keys of that assignment to all eternity. (D&C 128: 21.) They are expected to come to the great meeting when keys are returned to Adam and then, in turn, to Christ, preliminary to His return as the One whose right it is to preside over all things. (TPJS p. 157.)
I suppose the best way to be invited to that meeting would be to obtain a key from the Lord, perform in strict conformity to the assignment He gives you, and become thereby entitled to return that key in the great assembly.