Scheduled maintenance

•
New configuration:


  • 312 compute nodes
  • 9984 processing elements
  • /work - 175TB
  • /shared - 217TB
  • SLURM scheduler
  • UiB usernames for UiB users
Please find below a short list of changes:

  1. All users (except IMR) have to reaply for access at https://skjemaker.app.uib.no/view.php?id=2901837
  2. SLURM is a new job scheduler
    1. Documentation link https://docs.hpc.uib.no/wiki/Job_execution_(Hexagon)
    2. External Torque/Moab to Slurm reference https://www.glue.umd.edu/hpcc/help/slurm-vs-moab.html
  3. Please use Support for help and support.
  4. The hexagon@hpc.uib.no mailing list will be migrated to a self managed mailing list in a short time. All current mailing list users will be removed soon. If you want to subscribe please get back to our syslog  https://syslog.hpc.uib.no in a week, we will post a link to the new mailing list.

We didn’t manage to replace all HW components we’ve planned during this maintenance. We are planning to have a shorter maintenance somewhere in winter/spring to finish this job. 

All software as modules is still available, we will review and remove old in the coming weeks.

There are some major changes, as the new job scheduler and the HW configuration, maybe some things stopped working for you, some configurations are not finally in place, we will continue on improving this as well as updating documentation during the following weeks, we ask for your patience. And of course all feedback is welcome at support@hpc.uib.no.

The following changes will come in the next months:

  • /shared will be bigger in a few weeks
  • Bigger /home after the next maintenance window

•
There is a planned maintenance on the electric power line in the server room for the 25th of September. Therefore Hexagon, related file systems and storage enclosures has to be taken offline.

The maintenance is scheduled to start at 20:00. According to plan, Hexagon should be back by the end of the day.

Running jobs will be stopped. All scheduled jobs in the queue will be started automatically when the system is operational again.

Update 20.09.2017: Please note the time change. The maintenance window has been moved from Saturday, 23rd of September to Monday, 25th of September 20:00.

Update 26.09.2017: UPS maintenance is over yesterday night, and we have a problem to take hexagon online due to some filesystem storage issues. We are working on it and we apologize for inconveniences.

Update 26.09.2017:  Hexagon is up and available again since 09:08 AM.

Update 14:00_26.09.2017: Hexagon work file system crashed unexpectedly, we are working on it. sorry for inconveniences.

Update 14:40_26.09.2017 Hexagon has to be taken down due to hardware issues related to work filesystem. We try to resolve problems as soon as possible.

Update 16:40_26.09.2017 Hexagon is back online again. problem with work filesystem is resolved.

•
On Friday, 25th of August maintenance on electric lines in the server room will be carried out. Therefore Hexagon must be switched off.  All related file systems (/work, /work-common) will be also off.

The maintenance will start at 07:00 and according to the plan should last until 13:00 o'clock.

During this time work-common will not be available on  Grunch .

Update:
  • 25.08.2017 07:00: Maintenance has started.
  • 25.08.2017 12:50: Storage controller issues are delaying startup of the machine. We are working on the fix.
  • 25.08.2017 15:05: Storage controller issues were remediated. Some disks are rebuilding for /work-common filesystem, thus performance impact might be expected for a couple of days.
  • 25.08.2017 15:20: Hexagon is up again.

•
/work and /work-common filesystems will be unavailable on Grunch on 18th of October starting from 09:00 o'clock. This downtime is part of the scheduled maintenance advertised at
http://syslog.hpc.uib.no/2016/09/21/hexagon-planned-maintenance-18-10-19-10/.

Length of downtime is up to 8 hours for /work-common and up to 2 days for /work.

Please make sure that by this time there are no jobs using /work or /work-common, to avoid data-loss and/or data corruption.

We will keep you updated here.

Update: 2016-10-19 11:07 /work-common is back online and re-mounted on grunch.

•
We will have a two day planned maintenance on hexagon starting on 18th of October 09:00.

During the maintenance we will carry out filesystem upgrade, firmware upgrades as well service the hardware.

The job submission system has reservation in place, thus jobs which are not able to finish before maintenance start, will not be started.


Update: 2016-10-18 09:35 Maintenance has started, slightly delayed due to traffic jam.

Update: 2016-10-19 12:00 Maintenance has been finished, Hexagon is up and accessible again.

•
There is a scheduled maintenance on UPS and UPS power lines in HPC server room on Saturday, 20th Feb. All HPC resources will be stopped at 8:30, we are expecting this maintenance to finish before 17:00 same day.

Hexagon, FImm, Grunch and other connected to them resources will be unavailable. Hexagon queuing system has reservation in place, so that jobs which are not able to finish before the maintenance will not be started.

Update:
2016-02-20 07:45 System maintenance has started.
2016-02-20 16:30 /work-common filesystem storage got damaged, recovery progress is ongoing.
2016-02-21 07:30 System maintenance has finished, HPC systems are functional again.

•

During the maintenance we have:

  • applied different firmware updates and patches
  • installed newer libraries, compilers and tools

Please note that all libraries compiled with previous version of PGI will have to be recompiled.

Below you will find the complete list of the newly installed software:

  • CCE 8.4.0
  • Chapel 1.12.0
  • Craype 2.4.2
  • GCC 5.1.0
  • FFTW 3.3.4.5
  • HDF5 1.8.14
  • PGI 15.3.0
  • PerfTools 6.3.0
  • MPI 7.2.5
  • NetCDF 4.3.3.1
  • Totalview 8.15.7