Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

VASP compilation

News

VASP

Books

Recommended Links

 VASP Performance optimization

Intel Composer XE BLAS and LAPACK routines
Message Passing Interface MPI performance optimization     DDR Memory DDR3-1866 Memory Performance  Intel CPUs Price/Performance Ratio
Infiniband Installing Mellanox InfiniBand Driver on RHEL 6.5 ulimit problem with infiniband       GPFS on Red Hat
Oracle Grid Engine            
NFS performance tuning nfsstat Unix System Monitoring More cores per node is not necessarily better Admin Horror Stories Humor  Etc

Intel compiler installation

Intel has the most idiotic Web site I ever encounter among software vendors.  It is very difficult to get to the download even if you have the license.  You need specific version of compiler and libraries:

You need at least Intel-Composer-XE  (which is essentially Fortran compiler, C++ compiler  + libraries bundle). See comparison table at  https://software.intel.com/en-us/intel-xe-product-comparison.

You can also use gcc and use C preprocessor from gcc to compile VASP but this does not makes much sense as performance of OpenMPI might well be critical.  

Purchase Options:  Intel® Cluster Studio XE Intel® Parallel Studio XE Intel® C++ Studio XE Intel® Fortran Studio XE Intel® Composer XE Intel® C++ Composer XE Intel® Fortran Composer XE
Intel® C / C++ Compiler Y Y Y   Y Y  
Intel® Fortran Compiler Y Y   Y Y   Y
Intel® Math Kernel Library3 Y Y Y Y Y Y Y

If you bought single user license, you generally are better off installing compiler under the user you plan to use vasp, for example vasp.

I think two best recommendation for vasp compilation are:

Vaspwiki recommendations

Installing VASP - Vaspwiki

For the parallel compilation of VASP the following software is mandatory:

Intel Composer XE (12.1.3)

When installing the Intel Composer XE software package, both the Intel Fortran compiler ifort (version 12.1.3) as well as the Intel Math Kernel Library (MKL) in version 10.3 is installed.

The Intel MKL includes libraries for BLAS, LAPACK ScaLAPACK and their own implementation of FFT. No additional numerical libraries have to be compiled and installed. Make sure that the following file is sourced in your ~/.bashrc:

source /opt/intel/composerxe/bin/compilervars.sh [ia32|intel64]

where you should use ia32 for 32bit and intel64 for 64bit operating systems.

Intel MKL FFTW wrappers

VASP internally uses the calling statements of the FFTW3 which differ to the implementation of the FFT in the MKL. Therefore we have to build wrapper routines which are shipped with the MKL but not compiled:

cd $MKLROOT/interfaces/fftw3xf/
make

After a successful compilation libfftw3xf_intel.a was created in this directory.

openMPI

Download a version of openMPI (we use version 1.4.5) and untar the tarball.

 tar -xjvf openmpi-1.4.5.tar.bz2
 cd openmpi-1.4.5

Use

 ./configure --help

to get help for all the compilation options available. We used the following configure line, ensuring the Intel Fortran and C compilers are used

 ./configure CC=icc CXX=icpc F77=ifort FC=ifort --prefix=PREFIX

and where PREFIX is the absolute path to the directory where it should be installed on your system. Use the following naming for the final directory to know which Fortran compiler was used opempi-1.4.5-INTEL-12.1.3. After configuration has finished execute:

 make
 make install 

Probably you need root privileges to execute the last command. Finally we have to make the openMPI directories available for your bash. Therefore we include the following statement in the ~/.bashrc

 if [ -f /opt/intel/composerxe/bin/compilervars.sh ]
 then
   source /opt/intel/composerxe/bin/compilervars.sh intel64 2> /dev/null
   export MPI=[PREFIX]
   export PATH=$MPI/bin:$PATH
   export LD_LIBRARY_PATH=$MPI/lib:$LD_LIBRARY_PATH
   export MANPATH=$MANPATH:$MPI/share/man
   export INFOPATH=$INFOPATH:$MPI/share/man
 fi

In the first line we test whether the Intel composer is installed by checking whether the compilervars.sh file exists. Then we source the appropriate compiler and the openMPI version compiled with this compiler. PREFIX is the directory you installed openMPI.

Compilation

The following flags should be changed in the makefile, so that the most recent version of VASP compiles.

FC 

We are using openMPI for the parallel MPI environment and therefore we have to compile with the MPI version of the Fortran compiler mpif90. Mind that the used Fortran compiler has to be identical to the compiler used to compile openMPI.

FC=mpif90 
FCL 

The -mkl option tells the ifort compiler to link all necessary MKL libraries. Mind that this uses the ordinary LAPACK routines and not ScaLAPACK. Therefore you can comment all BLAS, LAPACK, BLACS, SCA statements in the makefile.

FCL=mpif90 -mkl

FFLAGS

Some namings of compiler flags changed in the most recent version of the Intel Fortran compiler. This is optional, but will prevent warnings and in the future it has to be changed since the old flags are deprecated.

FFLAGS =  -free -names lowercase -assume byterecl

FFTW

Include the Intel MKL FFT wrapper library which we compiled at the beginning of this section.

FFT3D   = fftmpiw.o fftmpi_map.o  fftw3d.o  fft3dlib.o   $(MKLROOT)/interfaces/fftw3xf/libfftw3xf_intel.a

SCA

If one wants to use ScaLAPACK routines instead of LAPACK the following line will activate ScaLAPACK support in VASP and includes the necessary libraries. Mind that you have to recompile certain files to add support for ScaLAPACK.

SCA= -lmkl_scalapack_lp64.a -lmkl_blacs_openmpi_lp64

Intel© Math Kernel Library Link Line Advisor

A very handy site is the Intel© Math Kernel Library Link Line Advisor. With its help you get a recommendation for the best libraries which you should use on your particular system.

Add profiling support

To add profiling support in the most recent version of VASP you have to add -DPROFILING to the preprocessor flags under CPP

 CPP = $(CPP_) -DMPI ...... -DPROFILING

Afterwards execute

 make clean
 make vasp

to recompile all source files of VASP. For further reading see Profiling

Hybrid openMPI/openMP parallelization

To enable explicit openMP support in VASP add -openmp to the FC line in the makefile

FC=mpif90 -openmp

Since this subject is more complex see Hybrid openMPI/openMP parallelization fur further information

Peter Larsson Notes

Here is a copy of Peter Larsson notes which looks like the best and the most current from many similar notes on the WEB, see original at How to compile VASP on NSC's Triolith  Jan 9th, 2013. the site contains additional interesting material and is highly recommended.  You can compare Peter Larsson's recommendations with  Cluster installation of VASP with Intel Compilers by Lev Lafayette

... You need both the regular VASP source code, and the supporting “vasp 5” library:

vasp.5.3.3.tar.gz
vasp.5.lib.tar.gz

I suggest to make a new directory called e.g. vasp.5.3.3, where you download and expand them. You would type commands approximately like this:

mkdir 5.3.3
cd 5.3.3
(download)
tar zxvf vasp.5.3.3.tar.gz
tar zxvf vasp.5.lib.tar.gz

Currently, you want to load these modules:

intel/12.1.4
impi/4.0.3.008
mkl/10.3.10.319

Which you can get bundled in the following module:

module load build-environment/nsc-recommended
VASP 5 lib

Compiling the VASP 5 library is straightforward. It contains some timing and IO routines, necessary for VASP, and LINPACK. My heavy edited makefile looks like this:

.SUFFIXES: .inc .f .F
#-----------------------------------------------------------------------
# Makefile for VASP 5 library on Triolith
#-----------------------------------------------------------------------

# C-preprocessor
CPP     = gcc -E -P -C -DLONGCHAR $*.F >$*.f
FC= ifort

CFLAGS = -O
FFLAGS = -Os -FI
FREE   =  -FR

DOBJ =  preclib.o timing_.o derrf_.o dclock_.o  diolib.o dlexlib.o drdatab.o


#-----------------------------------------------------------------------
# general rules
#-----------------------------------------------------------------------

libdmy.a: $(DOBJ) linpack_double.o
    -rm libdmy.a
    ar vq libdmy.a $(DOBJ)

linpack_double.o: linpack_double.f
    $(FC) $(FFLAGS) $(NOFREE) -c linpack_double.f

.c.o:
    $(CC) $(CFLAGS) -c $*.c
.F.o:
    $(CPP) 
    $(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f
.F.f:
    $(CPP) 
.f.o:
    $(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f

Note the addition of the “-DLONGCHAR” flag on the CPP line. It activates the longer input format for INCAR files, e.g. you can have MAGMOM lines with more than 256 characters. Now compile the library with the “make” command and check that you have the “libdmy.a” output file. Leave the file here, as the main VASP makefile will include it directly from here.

VASP 5 binary

Preparations

I only show how to build the parallel version with MPI and SCALAPACK here, as that is what you should run on Triolith. Navigate to the “vasp.5.3” library where the main source code is:

cd ..
cd vasp.5.3

Before we start, we want to think about how to find the external libraries that we need. These are:

For BLAS/LAPACK, we are going to use Intel’s Math Kernel Library (“MKL” henceforth). The easiest way to link to MKL at NSC is by adding the two following flags to the compiler command:

ifort -Nmkl -mkl=sequential ...

For fast Fourier transforms, we could use the common FFTW library with VASP, but MKL actually contains its own optimized FFTs together with an FFTW interface, so we can use these instead. Provided that we link with MKL, which we are already doing in order to get BLAS/LAPACK, we do not need to do anything more. The linker should pick up the FFTW subroutines automatically.

For MPI, we are going to use Intel’s MPI library. We have already loaded the “impi/4.0.3.008” module, so all we have to do is to add the “-Nmpi” flag to compiler command:

ifort -Nmpi ...

We don’t need to add explicit paths to any MPI libraries, or use the special “mpif90” compiler wrapper.

Editing the makefile

I suggest that you start from the Linux/Intel Fortran makefile:

cp makefile.linux_ifc_P4 makefile

It is important to realize that the makefile is split in two parts, and is intended to be used in an overriding fashion. If you don’t want to compile the serial version, you should enable the definitions of FC, CPP etc in the second half of the makefile to enable parallel compilation. These will then override the settings for the serial version.

Start by editing the Fortran compiler and its flags:

FC=ifort -I$(MKL_ROOT)/include/fftw 
FFLAGS =  -FR -lowercase -assume byterecl -Nmpi 

We need to add “-Nmpi” to get proper linking with Intel MPI at NSC. Then, we change the optimization flags:

OFLAG=-O2 -ip -xavx 

This is to be on the safe side, so that we get AVX optimizations. Include MKL with FFTW like this:

BLAS = -mkl=sequential
LAPACK = 

We use the serial version of MKL, without any multithreading, as VASP runs MPI on all cores with great success. Set the NSC specific linking options for MKL and MPI:

LINK    = -Nmkl -Nmpi 

Uncomment the CPP section for the MPI parallel VASP:

CPP    = $(CPP_) -DMPI  -DHOST=\"LinuxIFC\" -DIFC \
     -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \
     -DMPI_BLOCK=8000 -Duse_collective -DscaLAPACK
    -DRPROMU_DGEMV  -DRACCMU_DGEMV

Change it to something like this:

CPP     = $(CPP_) -DMPI -DHOST=\"TRIOLITH-BUILD01\" -DIFC \
          -DCACHE_SIZE=4000  -DPGF90 -Davoidalloc -DNGZhalf \
          -DMPI_BLOCK=262144 -Duse_collective -DscaLAPACK \
          -DRPROMU_DGEMV  -DRACCMU_DGEMV  -DnoSTOPCAR

CACHE_SIZE is only relevant for Furth FFTs, which we do not use. The HOST variable is written out in the top of the OUTCAR file. It can be anything which helps you identify this compilation of VASP. The MPI_BLOCK variable needs to be set higher for best performance on Triolith. And finally, “noSTOPCAR” will disable the ability to stop a calculation by using the STOPCAR file. We do this to improve file I/O against the global file systems. (Otherwise, each VASP process will have to check this file for every SCF iteration.)

Finally, we enable SCALAPACK from MKL:

SCA= -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64

And the parallelized version of the fast Fourier transforms with FFTW bindings:

FFT3D   = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

Note that we do not need to link to FFTW explicitly, since it is included in MKL. Finally, we uncomment the last library section:

LIB     = -L../vasp.5.lib -ldmy  \
      ../vasp.5.lib/linpack_double.o \
      $(SCA) $(LAPACK) $(BLAS)

We have to do this to include the “$(SCA)” variable. The full makefile can be found here on Triolith:

/software/apps/vasp/5.3.3-18Dec12/build01/makefile
Compiling

VASP does not have a makefile that supports parallel compilation. So in order to compile we just do:

make

... ... ...

When finished, you should find a binary called “vasp”.

Running

When you compile according to these instructions, there is no need to set LD_LIBRARY_PATHs and such. Instead, the ifort compiler will hard-code all library paths by using the RPATH mechanism and write information into the binary file about which MPI version you used. This means that you can launch VASP directly like this in a job shell:

mpprun /path/to/vasp

Mpprun will automatically pick up the correct number of processor cores from the queue system and launch your vasp binary using Intel’s MPI launcher.

Posted by Peter Larsson Jan 9th, 2013


Top updates

Bulletin Latest Past week Past month
Google Search


NEWS CONTENTS

Old News ;-)

Building VASP with Intel® MKL and Intel® Compilers by Vipin Kumar E K...

| software.intel.com

This guide is intended to help users on how to build VASP (Vienna Ab-Initio Package Simulation) using Intel® Math Kernel Library (Intel® MKL) and Intel® Compilers and Intel® MPI.

Building VASP* with Intel® MKL and Intel® Compilers

Step 1 - Overview

This guide is intended to help users on how to build VASP (Vienna Ab-Initio Package Simulation) using Intel® Math Kernel Library (Intel® MKL) and Intel® Compilers and Intel® MPI.

VASP is a package for performing ab-initio quantum-mechanical molecular dynamics (MD) using pseudo potentials and a plane wave basis set. The approach implemented in VAMP/VASP is based on a finite-temperature local-density approximation (with the free energy as variational quantity) and an exact evaluation of the instantaneous electronic ground state at each MD-step using efficient matrix diagonalization schemes and an efficient Pulay mixing. These techniques avoid all problems occurring in the original Car-Parrinello method which is based on the simultaneous integration of electronic and ionic equations of motion. The interaction between ions and electrons is described using ultrasoft Vanderbilt pseudopotentials (US-PP) or the projector augmented wave method (PAW). Both techniques allow a considerable reduction of the necessary number of plane-waves per atom for transition metals and first row elements. Forces and stress can be easily calculated with VAMP/VASP and used to relax atoms into their instantaneous groundstate. [Ref: VASP]

Version Information

This application note was created to help users who benchmark clusters using VASP to also incorporate the latest version of Intel® MKL on Linux platforms on Xeon systems. Specifically, we'll address Intel® MKL version 11.1

More information on VASP can be found from http://cms.mpi.univie.ac.at/vasp/

Step 2 - Downloading VASP Source Code

VASP is not public-domain or share-ware, and will be distributed only after a license contract has been signed. Please visit VASP homepage to know more details on obtaining the license.

Prerequisites

  1. Intel® MKL contains highly optimized FFT and also the wrappers for FFTW, which can be obtained from the following options:
    • Download a FREE evaluation version of the Intel® MKL product.
    • Download the FREE non-commercial* version of the Intel® MKL product.
    • Purchase a copy of Intel® MKL

All of these can be obtained at: http://www.intel.com/software/products/mkl.

  1. Intel® C++ and FORTRAN compilers can be obtained from http://www.intel.com/softwre/products/compilers.
  1. Intel MPI can be obtained from http://www.intel.com/software/products/cluster.

Open source MPI (MPICH2) can be obtained from http://www-unix.mcs.anl.gov/mpi/mpich/ .

Step 3 - Configuration

Use the following commands to extract the VASP files:

view sourceprint?
1 $tar -xvzf vasp.tgz vasp.lib.tgz

This will create vasp and vasp.lib directories

Make sure that MPI, C++ and FORTRAN compilers are installed and in the PATH. Also set LD_LIBRARY_PATH to your compiler (C++ and FORTRAN), MPI, and Intel® MKL libraries.

Note: This application note is written specifically for use with the Intel compilers and MPI.

Step 4 - Building VASP

  1. Build FFTW wrapper library for Intel® MKL

Change directory to <your MKL installation>/interfaces/fftw3xf

Assuming default installation

view sourceprint?
1 $cd /opt/intel/mkl/interfaces/fftw3xf

Build the fftw 3.x FORTRAN wrapper library for Intel® 64 (previously known as em64t) platform using the Intel compilers.

view sourceprint?
1 $make libintel64 compiler=intel

This will create the fftw wrapper library “libfftw3xf_intel.a” in lib/em64t directory of your Intel® MKL installation.

  1. Build libdmy.a

Change directory to vasp.x.x.lib

Modify the makefile.linux_ifc_P4 Makefile to point to the correct Intel FORTRAN compiler.

view sourceprint?
1 FC=ifort

Run the following command from vasp.x.x.lib, using the Makefile for linux using the Intel compiler.

view sourceprint?
1 $make -f makefile.linux_ifc_P4

After a successful compilation, libdmy.a will be built in the same directory.

  1. Build VASP

Change directory to vasp.x.x

Export the environment variable MPIFC to point to the Intel MPI Fortran compiler mpiifort.

view sourceprint?
1 $export MPIFC=mpiifort

Edit the makefile.linux_ifc_P4 to link with Intel® MKL libraries and change the Fotran compiler

Under FORTRAN compiler and linker part of the makefile

view sourceprint?
1 FC=$(MPIFC)
view sourceprint?
1 CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \
2 -DCACHE_SIZE=32000 -DPGF90 -Davoidalloc -DNGZhalf \
3 -DMPI_BLOCK=64000 -DscaLAPACK -Duse_collective -Ddo_loops -DMKL_ILP64

Change the FORTRAN flags section as shown here.

view sourceprint?
1 FFFLAGS = -FR -names lowercase -assume byterecl -I$(MKLROOT)/include/fftw -xAVX
view sourceprint?
01 MKLROOT=/opt/intel/composer_xe_2013_sp1/mkl
02 MKL_PATH=$(MKLROOT)/lib/intel64
03 MKL_FFTW_PATH=$(MKL_PATH)/interfaces/fftw3xf
04
05 BLAS= -mkl=cluster
06
07 LAPACK= -mkl=cluster
08
09
10 .……
11 …………

Comment out the existing FFT3D line

view sourceprint?
1 #FFT3D = fftdfurth.o fftdlib.o
view sourceprint?
1 FFT3D= fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o $(MKL_FFTW_PATH)/libfftw3xf_intel.a
2
3 INCS = -I$(MKLROOT)/include/fftw

To use Intel® MKL ScaLAPACK, modify additional options for CPP for parallel version adding

-DscaLAPACK to the CPP line as mentioned above and specify Intel® MKL ScaLAPACK as below.

view sourceprint?
1 SCA=$(MKL_PATH)/libmkl_scalapack_ilp64.a $(MKL_PATH)/libmkl_intelmpi_ilp64.a

Run the following command to build vasp

view sourceprint?
1 $make -f makefile.linux_ifc_P4

This will create the VASP executable in the current directory.

Step 5 - Running VASP

Run vasp by executing mpiexec command with your required parameters. Below, for e.g., to run 32 processes use as shown with your workloads.

view sourceprint?
1 $mpiexec -n 32 ./vasp

Appendix B - Known Issues and Limitations

There could be a compilation error on nonlr.F file as reported below.

view sourceprint?
01 mpif90 -fc=ifort -FR -names lowercase -assume byterecl -O2 -ip -I/opt/intel/compiler/2013_sp1.1.106/composer_xe_2013_sp1.1.106/mkl/include/fftw -c nonlr.f90
02
03 nonlr.F(3069): error #6404: This name does not have a type, and must have an explicit type. [LM]
04 DO LM=1,LMMAXC
05 nonlr.F(3069): error #6404: This name does not have a type, and must have an explicit type. [LM]
06 DO LM=1,LMMAXC
07 ----------------^
08 nonlr.F(3069): error #6063: An INTEGER or REAL data type is required in this context. [LM]
09 DO LM=1,LMMAXC
10 ----------------^
11 nonlr.F(3206): error #6404: This name does not have a type, and must have an explicit type. [LM]
12 DO LM=1,LMMAXC
13 ----------------^
14 nonlr.F(3206): error #6063: An INTEGER or REAL data type is required in this context. [LM]
15 DO LM=1,LMMAXC
16 ----------------^

By applying the patch mentioned below to the nonlr.F, this compilation error can be fixed.

view sourceprint?
1 3002c3002
2 < INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, LM, IND
3 ---
4 > INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, IND
5 3144c3144
6 < INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, LM, IND
7 ---
8 > INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, IND

Appendix C - References

VASP (Vienna Ab-Initio Package Simulation)

Intel® Math Kernel Library

Intel® Cluster Studio XE

How to compile VASP on NSC's Triolith - Peter Larsson

This is the best guide for compilation of VASP that I have found on Internet. Partially reproduced above. Peter Larson uses intel MPI with VASP.

[Mar 21, 2014] Cluster installation of VASP with Intel Compilers by Lev Lafayette

Submitted by lev_lafayette on Thu, 02/13/2014 - 05:25

Previous comments concerning VASP installs still largely apply.

  1. It still performs ab-initio quantum-mechanical molecular dynamics (MD) using pseudopotentials and a plane wave basis set.
  2. It still has an weird and frustrating license which is open source (if you pay them) but not free.
  3. It still is released without standard autoconfiguration tools and with make files for a variety of system architectures which rarely match was diverse users might wish to use, thus requiring posts like this every time a new version is released.

In this instance we start of with a copy of VASP in the user's home directory. Make a directory called vasp (mkdir vasp) and and extract the tarball of the two VASP files there (tar xvf vasp.5.lib, tar xvf vasp.5.3).

# tar xvf vasp.5.lib.tar.gz
# tar xvf vasp.5.3.3.tar.gz
# module load openmpi-intel
# module load fftw/3.3.2-intel
# cd vasp.5.lib
# cp makefile.linux_ifc_P4 makefile
# vim makefile

Modify to the following (seriously guys, ifc?):

# diff makefile makefile.linux_ifc_P4
19c19
< FC=ifort
---
> FC=ifc

And install..

# make

Change to VASP

# cd ../vasp.5.3
# cp makefile.linux_ifc_P4 makefile
# vim makefile

Modify to the following:

diff makefile makefile.linux_ifc_P4
62c62
< FC=mpif90
---
> FC=ifort
149,151c149
< #BLAS= -lguide -mkl
<
< BLAS= -mkl
---
> BLAS= -lguide -mkl
160,162c158
< #LAPACK= $(MKL_PATH)/libmkl_intel_lp64.a
<
< LAPACK= /usr/local/intel/composerxe/composer_xe_2013_sp1.0.080/mkl/lib/intel64/libmkl_intel_lp64.a
---
> LAPACK= $(MKL_PATH)/libmkl_intel_lp64.a
209,210c205,206
< FC=mpif90
< FCL=$(FC)
---
> #FC=mpif90
> #FCL=$(FC)
227,230c223,226
< CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \
< -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \
< -DMPI_BLOCK=8000 -Duse_collective -DscaLAPACK
< # -DRPROMU_DGEMV -DRACCMU_DGEMV
---
> #CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \
> # -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \
> # -DMPI_BLOCK=8000 -Duse_collective -DscaLAPACK
> ## -DRPROMU_DGEMV -DRACCMU_DGEMV
245,247c241,243
< LIB = -L../vasp.5.lib -ldmy \
< ../vasp.5.lib/linpack_double.o \
< $(SCA) $(LAPACK) $(BLAS)
---
> #LIB = -L../vasp.5.lib -ldmy \
> # ../vasp.5.lib/linpack_double.o \
> # $(SCA) $(LAPACK) $(BLAS)
257,258c253
< FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o /usr/local/fftw/3.3.2-intel/lib/libfftw3.a
< # /opt/libs/fftw-3.1.2/lib/libfftw3.a
---
> #FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o /opt/libs/fftw-3.1.2/lib/libfftw3.a

Damned if I could find the guide libraries. Note the hardcoded path (ugh). I could recompile Intel...

.. and install

# make

The code should now compile and generate a 'vasp' executable, which the user should then be able to test, once changed to their user and group. Don't forget the OMP_NUM_THREADS issue previously cited.

Initial executable worked only in serial. Still fighting this and will update as progress continues. "A luta continua", as the Portuguese say.

Linking applications with Intel® MKL version 10.x Intel® Developer Zone

Intel® Math Kernel Library (Intel® MKL) 11.0 Intel® Developer Zone

Intel® Math Kernel Library (Intel® MKL) 11.0 includes a wealth of routines to accelerate application performance and reduce development time. Today’s processors have increasing core counts, wider vector units and more varied architectures. The easiest way to take advantage of all of that processing power is to use a carefully optimized computing math library designed to harness that potential. Even the best compiler can’t compete with the level of performance possible from a hand-optimized library.

Using Intel® MKL in VASP

Linux Toolkit


How to compile VASP - Micro and Nano Mechanics Group

The following table compares the time to run the same benchmark case as above using both executables. Our executable shows speed up with multiple CPUs.

Number of CPUs vasp compiled here /share/apps/vasp.4.6/vasp vasp another build (below)
1 76 (seconds) 75 (seconds) 48 (seconds)
2 59 (seconds) 72 (seconds) 34 (seconds)
4 35 (seconds) 64 (seconds) 29 (seconds)
8 37 (seconds) 65 (seconds) 31 (seconds)

... ... ...

This will create executable vasp in this directory. This time the executable vasp can not run interactively, but can only run in the queue through a PBS script (e.g. mpiexec --comm=ib -np 4 vasp). Notice that here we need to specify the communication channel. (For mvapich1 we need to use --comm=ib and for mvapich2 use --comm=pmi.)

The following table provides the timing information for the same benchmark case studied above.

Number of CPUs vasp compiled here
1 40 (seconds)
2 28 (seconds)
4 27 (seconds)
6 29 (seconds)
8 31 (seconds)
16 215 (seconds)

A Word of Caution: Make sure to run a few test cases to confirm your executable not only runs but produces the correct numerical results. For example, we have found that on su-ahpcrc, function BRMIX (broyden.f) was giving serious errors. This was solved by changing the compilation options to

"OFLAG=-O1 -mtune core2 -axW -unroll",

and by changing "ICHARG = 0" to "ICHARG = 2" in INCAR. ("ICHARG=2" is the default when "ISTART=0" or if there are no CHG, CHGCAR, WAVECAR files in the folder.)

... ... ...

Intel MKL instructions

There are "official" instructions to compile VASP with the Intel Compiler family, named Using Intel MKL in VASP. Those instructions are unrelated with the present instructions but can be a good reference for future builds.

linux ifort compile VASP, fortcom Severe Internal compiler error Intel® Developer Zone

I used the ifort 10.1.018 version to compile the VASP(a chemical soft).

All the parameters were set up as following(in the makefile):

Makefile:

#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=ifort
# fortran linker
FCL=$(FC)

OFLAG=-O1 -xW -tpp7

OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =

OBJ_NOOPT =
DEBUG = -FR -O0
INLINE = $(OFLAG)

#-----------------------------------------------------------------------
# the following lines specify the position of BLAS and LAPACK
# on P4, VASP works fastest with the libgoto library
# so that's what I recommend
#-----------------------------------------------------------------------

# Atlas based libraries
#ATLASHOME=/opt/Linux_P4SSE2/lib
#ATLASHOME= $(HOME)/archives/BLAS_OPT/ATLAS/lib/Linux_P4SSE2/
#BLAS= /opt/Linux_P4SSE2/lib -lf77blas -latlas -lmkl
#BLAS= -L$(ATLASHOME) -lf77blas -latlas

# use specific libraries (default library path might point to other libraries)
#BLAS= $(ATLASHOME)/libf77blas.a $(ATLASHOME)/libatlas.a

# use the mkl Intel libraries for p4 (www.intel.com)
# mkl.5.1
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl_p4 -lpthread

# mkl.5.2 requires also to -lguide library
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
BLAS= /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl -lguide -lpthread

# even faster Kazushige Goto's BLAS
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
#BLAS= /opt/libs/libgoto/libgoto_p4_512-r0.6.so

# LAPACK, simplest use vasp.4.lib/lapack_double
LAPACK= ../vasp.4.lib/lapack_double.o

# use atlas optimized part of lapack
#LAPACK= ../vasp.4.lib/lapack_atlas.o -llapack -lcblas

# use the mkl Intel lapack
#LAPACK= -lmkl_lapack
......

......

Output:

............

............

ifort -FR -lowercase -O1 -tpp7 -xW -prefetch- -prev_div -unroll0 -vec_report3 -c fft3dlib.f90
ifort: command line remark #10148: option '-tp' not supported
ifort: command line warning #10156: ignoring option '-p'; no argument required
ifort -FR -lowercase -FR -O0 -c main.f90
rm -f vasp
ifort -o vasp main.o base.o mpi.o smart_allocate.o xml.o constant.o jacobi.o main_mpi.o scala.o asa.o lattice.o poscar.o ini.o setex.o radial.o pseudo.o mgrid.o mkpoints.o wave.o wave_mpi.o symmetry.o symlib.o lattlib.o random.o nonl.o nonlr.o dfast.o choleski2.o mix.o charge.o xcgrad.o xcspin.o potex1.o potex2.o metagga.o constrmag.o pot.o cl_shift.o force.o dos.o elf.o tet.o hamil.o steep.o chain.o dyna.o relativistic.o LDApU.o sphpro.o paw.o us.o ebs.o wavpre.o wavpre_noio.o broyden.o dynbr.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o brent.o stufak.o fileio.o opergrid.o stepver.o dipol.o xclib.o chgloc.o subrot.o optreal.o davidson.o edtest.o electron.o shm.o pardens.o paircorrection.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o setlocalpp.o fft3dfurth.o fft3dlib.o -L../vasp.4.lib -ldmy ../vasp.4.lib/linpack_double.o ../vasp.4.lib/lapack_double.o /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
backend signals

fortcom: Severe: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.

ifort: error #10014: problem during multi-file optimization compilation (code 3)make: *** [vasp] Error 3

========================================

I have no idea whats wrong with my compilation..

if anyone can help me figure it out.. My email is xz288@nyu.edu

Thanks.

For more complete information about compiler optimizations, see our Optimization Notice.

Kevin Davis (Intel)

Wed, 10/22/2008 - 05:21

The backend component of the compiler suffered an internal compilation error. Since you are already using the latest 10.1.018 update but only a few optimization switches, you could try the first work-around of dropping the optimization from O1 to O0 in OFLAG. If that avoids the error, then use a trial and error search to see if you can identify which source file corresponding to the objects linked triggers the error with compiled at O1. It is also possible the error is triggered by xW so you could do a similar search involving that option. You probably noted the tpp7 is an invalid option, so you can remove it.

If the source for VASP can be provided to us then please report the error to Intel Premier here and indicate that you reported this error in the User forum.

You could also consider trying the 11.0 Beta compiler. See the permanent announcement regarding that at the top of this Forum.

xqzhang

Wed, 10/22/2008 - 13:22

Quoting - Kevin Davis (Intel)

The backend component of the compiler suffered an internal compilation error. Since you are already using the latest 10.1.018 update but only a few optimization switches, you could try the first work-around of dropping the optimization from O1 to O0 in OFLAG. If that avoids the error, then use a trial and error search to see if you can identify which source file corresponding to the objects linked triggers the error with compiled at O1. It is also possible the error is triggered by xW so you could do a similar search involving that option. You probably noted the tpp7 is an invalid option, so you can remove it.

If the source for VASP can be provided to us then please report the error to Intel Premier here and indicate that you reported this error in the User forum.

You could also consider trying the 11.0 Beta compiler. See the permanent announcement regarding that at the top of this Forum.

Hi, thanks for your reply.. i tried your way, but still get the same error.

Then i switched to another pc, and changed alittle when linking the blas library by MKL as following:

ifort flag:

OFLAG=-O3 -xW -tpp7

BLAS= /opt/intel/mkl/10.0.5.025/lib/32 -lmkl_P4 -lguide -lpthrea

=============================================================

for this time, i got a different error:

===========================error=============================

ipo: error #11021: OBJREAD Error: Could not create mapping for /opt/intel/mkl/10.0.5.025/lib/32/
ifort: error #10014: problem during multi-file optimization compilation (code 1)make: *** [vasp] Error 1

============================================================

so,what should i do now?

thanks for your help. :)

Shane Story (Intel)

Wed, 10/22/2008 - 17:15

Quoting - xqzhang

Hi, all

I used the ifort 10.1.018 version to compile the VASP(a chemical soft).

All the parameters were set up as following(in the makefile):

Makefile:

#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=ifort
# fortran linker
FCL=$(FC)

OFLAG=-O1 -xW -tpp7

OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =

OBJ_NOOPT =
DEBUG = -FR -O0
INLINE = $(OFLAG)

#-----------------------------------------------------------------------
# the following lines specify the position of BLAS and LAPACK
# on P4, VASP works fastest with the libgoto library
# so that's what I recommend
#-----------------------------------------------------------------------

# Atlas based libraries
#ATLASHOME=/opt/Linux_P4SSE2/lib
#ATLASHOME= $(HOME)/archives/BLAS_OPT/ATLAS/lib/Linux_P4SSE2/
#BLAS= /opt/Linux_P4SSE2/lib -lf77blas -latlas -lmkl
#BLAS= -L$(ATLASHOME) -lf77blas -latlas

# use specific libraries (default library path might point to other libraries)
#BLAS= $(ATLASHOME)/libf77blas.a $(ATLASHOME)/libatlas.a

# use the mkl Intel libraries for p4 (www.intel.com)
# mkl.5.1
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl_p4 -lpthread

# mkl.5.2 requires also to -lguide library
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
BLAS= /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl -lguide -lpthread

# even faster Kazushige Goto's BLAS
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
#BLAS= /opt/libs/libgoto/libgoto_p4_512-r0.6.so

# LAPACK, simplest use vasp.4.lib/lapack_double
LAPACK= ../vasp.4.lib/lapack_double.o

# use atlas optimized part of lapack
#LAPACK= ../vasp.4.lib/lapack_atlas.o -llapack -lcblas

# use the mkl Intel lapack
#LAPACK= -lmkl_lapack
......

......

Output:

............

............

ifort -FR -lowercase -O1 -tpp7 -xW -prefetch- -prev_div -unroll0 -vec_report3 -c fft3dlib.f90
ifort: command line remark #10148: option '-tp' not supported
ifort: command line warning #10156: ignoring option '-p'; no argument required
ifort -FR -lowercase -FR -O0 -c main.f90
rm -f vasp
ifort -o vasp main.o base.o mpi.o smart_allocate.o xml.o constant.o jacobi.o main_mpi.o scala.o asa.o lattice.o poscar.o ini.o setex.o radial.o pseudo.o mgrid.o mkpoints.o wave.o wave_mpi.o symmetry.o symlib.o lattlib.o random.o nonl.o nonlr.o dfast.o choleski2.o mix.o charge.o xcgrad.o xcspin.o potex1.o potex2.o metagga.o constrmag.o pot.o cl_shift.o force.o dos.o elf.o tet.o hamil.o steep.o chain.o dyna.o relativistic.o LDApU.o sphpro.o paw.o us.o ebs.o wavpre.o wavpre_noio.o broyden.o dynbr.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o brent.o stufak.o fileio.o opergrid.o stepver.o dipol.o xclib.o chgloc.o subrot.o optreal.o davidson.o edtest.o electron.o shm.o pardens.o paircorrection.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o setlocalpp.o fft3dfurth.o fft3dlib.o -L../vasp.4.lib -ldmy ../vasp.4.lib/linpack_double.o ../vasp.4.lib/lapack_double.o /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
backend signals

fortcom: Severe: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.

ifort: error #10014: problem during multi-file optimization compilation (code 3)make: *** [vasp] Error 3

========================================

I have no idea whats wrong with my compilation..

if anyone can help me figure it out.. My email is xz288@nyu.edu

Thanks.

Shane Story (Intel)

Wed, 10/22/2008 - 17:19

Quoting - xqzhang

Hi, all

I used the ifort 10.1.018 version to compile the VASP(a chemical soft).

All the parameters were set up as following(in the makefile):

Makefile:

#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=ifort
# fortran linker
FCL=$(FC)

OFLAG=-O1 -xW -tpp7

OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =

OBJ_NOOPT =
DEBUG = -FR -O0
INLINE = $(OFLAG)

#-----------------------------------------------------------------------
# the following lines specify the position of BLAS and LAPACK
# on P4, VASP works fastest with the libgoto library
# so that's what I recommend
#-----------------------------------------------------------------------

# Atlas based libraries
#ATLASHOME=/opt/Linux_P4SSE2/lib
#ATLASHOME= $(HOME)/archives/BLAS_OPT/ATLAS/lib/Linux_P4SSE2/
#BLAS= /opt/Linux_P4SSE2/lib -lf77blas -latlas -lmkl
#BLAS= -L$(ATLASHOME) -lf77blas -latlas

# use specific libraries (default library path might point to other libraries)
#BLAS= $(ATLASHOME)/libf77blas.a $(ATLASHOME)/libatlas.a

# use the mkl Intel libraries for p4 (www.intel.com)
# mkl.5.1
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl_p4 -lpthread

# mkl.5.2 requires also to -lguide library
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
BLAS= /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl -lguide -lpthread

# even faster Kazushige Goto's BLAS
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
#BLAS= /opt/libs/libgoto/libgoto_p4_512-r0.6.so

# LAPACK, simplest use vasp.4.lib/lapack_double
LAPACK= ../vasp.4.lib/lapack_double.o

# use atlas optimized part of lapack
#LAPACK= ../vasp.4.lib/lapack_atlas.o -llapack -lcblas

# use the mkl Intel lapack
#LAPACK= -lmkl_lapack
......

......

Output:

............

............

ifort -FR -lowercase -O1 -tpp7 -xW -prefetch- -prev_div -unroll0 -vec_report3 -c fft3dlib.f90
ifort: command line remark #10148: option '-tp' not supported
ifort: command line warning #10156: ignoring option '-p'; no argument required
ifort -FR -lowercase -FR -O0 -c main.f90
rm -f vasp
ifort -o vasp main.o base.o mpi.o smart_allocate.o xml.o constant.o jacobi.o main_mpi.o scala.o asa.o lattice.o poscar.o ini.o setex.o radial.o pseudo.o mgrid.o mkpoints.o wave.o wave_mpi.o symmetry.o symlib.o lattlib.o random.o nonl.o nonlr.o dfast.o choleski2.o mix.o charge.o xcgrad.o xcspin.o potex1.o potex2.o metagga.o constrmag.o pot.o cl_shift.o force.o dos.o elf.o tet.o hamil.o steep.o chain.o dyna.o relativistic.o LDApU.o sphpro.o paw.o us.o ebs.o wavpre.o wavpre_noio.o broyden.o dynbr.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o brent.o stufak.o fileio.o opergrid.o stepver.o dipol.o xclib.o chgloc.o subrot.o optreal.o davidson.o edtest.o electron.o shm.o pardens.o paircorrection.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o setlocalpp.o fft3dfurth.o fft3dlib.o -L../vasp.4.lib -ldmy ../vasp.4.lib/linpack_double.o ../vasp.4.lib/lapack_double.o /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
backend signals

fortcom: Severe: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.

ifort: error #10014: problem during multi-file optimization compilation (code 3)make: *** [vasp] Error 3

========================================

I have no idea whats wrong with my compilation..

if anyone can help me figure it out.. My email is xz288@nyu.edu

Thanks.

For instructions on how to edit the VASP makefile to use MKL, see this application note:

http://www.intel.com/support/performancetools/libraries/mkl/sb/CS-028850.htm

Thanks, Shane

Zach Lukaszek

Tue, 05/29/2012 - 10:59

Dear future people: I actually just ran into this issue minutes ago while compiling Q-Chem 4.0 with parallel flags at the ohio supercomputing center. I was using intel-compilers-10.0.023 and updating to 11.1.056 fixed the problem immediately for me. I don't know if this is useful to anyone, but I figured there should be a record of it for future googlers.

Recommended Links

Softpanorama hot topic of the month

Softpanorama Recommended

Compilation

Forums

 



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: December 15, 2016