According to the privacy regulations of the health insurance portability and accountability act (HIPAA), patients' control over electronic health data is one of the major concern issues. Currently, remote access authorization is considered as the best solution to guarantee the patients' control over their health data. In this paper, a new biometric-based key management scheme is proposed to facilitate remote access authorization anytime and anywhere. First, patients and doctors can use their biometric information to verify the authenticity of communication partners through real-time video communication technology. Second, a safety channel is provided in delivering their access authorization and secret data between patient and doctor. In the designed scheme, the user's public key is authenticated by the corresponding biometric information without the help of public key infrastructure (PKI). Therefore, our proposed scheme does not have the costs of certificate storage, certificate delivery, and certificate revocation. In addition, the implementation time of our proposed system can be significantly reduced.