U.S. Defense Secretary Leon Panetta says U.S. is losing its patience with Pakistan.