TIL : tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize / Internal: ptxas exited with non-zero error code 256

1. tensorflow.python.framework.errors_impl.UnknownError:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize

 

cudnn : 7.6.5

tensorflow : 2.3.0

cuda : 10.2

 

 

 

해결법 1:

엔비디아 드라이버 버전을 최신으로 업데이트

 

sudo apt-get update

apt-cache search nvidia-driver

sudo apt-get install nvidia-418

sudo reboot

 

출처: https://eehoeskrap.tistory.com/290 [Enough is not enough]

 

해결법 2:

메모리 확인

nvidia-smi로 gpu 메모리 확인 시 남은 메모리가 없는 경우

ps -ef 로 필요없는 메모리 사용량을 확인하고 kill 

 

 

2. tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7

2021-01-04 10:30:50.900361: W tensorflow/stream_executor/gpu/asm_compiler.cc:81] Running ptxas --version returned 256

2021-01-04 10:30:50.955724: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 

Relying on driver to perform ptx compilation. 

Modify $PATH to customize ptxas location.

This message will be only logged once.

 

 

해결법 : 

Can confirm upgrading to CUDA toolkit 11.1 solved this for me.

 

 

출처: https://github.com/tensorflow/tensorflow/issues/40036