gistfile1.txt

o.d.e.m.MultiGpuLenetMnistExample - Build model....
o.d.e.m.MultiGpuLenetMnistExample - Train model....
CUDA error at /skymind/libnd4j/blas/cuda/NativeOps.cu:5907 code=77(<unknown>) "cudaStreamSynchronize(*stream)" 
o.d.p.ParallelWrapper - Averaged score: 2.079001024173573
Exception in thread "main" java.lang.RuntimeException: Can't allocate [HOST] memory: 32; threadId: 1
	at org.nd4j.jita.memory.impl.CudaDirectProvider.malloc(CudaDirectProvider.java:59)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.malloc(CudaCachingZeroProvider.java:113)
	at org.nd4j.jita.memory.impl.CudaFullCachingProvider.malloc(CudaFullCachingProvider.java:77)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.alloc(CudaZeroHandler.java:218)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.alloc(CudaZeroHandler.java:239)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.allocateMemory(AtomicAllocator.java:377)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.allocateMemory(AtomicAllocator.java:336)
	at org.nd4j.linalg.jcublas.buffer.BaseCudaDataBuffer.<init>(BaseCudaDataBuffer.java:143)
	at org.nd4j.linalg.jcublas.buffer.BaseCudaDataBuffer.<init>(BaseCudaDataBuffer.java:180)
	at org.nd4j.linalg.jcublas.buffer.CudaDoubleDataBuffer.<init>(CudaDoubleDataBuffer.java:55)
	at org.nd4j.linalg.jcublas.JCublasNDArrayFactory.average(JCublasNDArrayFactory.java:797)
	at org.nd4j.linalg.jcublas.JCublasNDArrayFactory.average(JCublasNDArrayFactory.java:848)
	at org.nd4j.linalg.factory.Nd4j.averageAndPropagate(Nd4j.java:4750)
	at org.deeplearning4j.parallelism.ParallelWrapper.fit(ParallelWrapper.java:148)
	at org.deeplearning4j.examples.multigpu.MultiGpuLenetMnistExample.main(MultiGpuLenetMnistExample.java:145)