Interesting topics

Table of content

Description

At the TF-NOC meeting during fall 2010 we agreed to collect information about what the TF-NOC participants thought would be interesting to share from their organisation, as well as information they would be interested in receiving from other organisations. The aim is to summarize that collected information on this page.

Summary of interesting topics

BELNET
What BELNET would like to present:

lessons learned from outsourced incident & change management
approach & tools: how the different departments & the NOC's work together
scripting & naming convention: making the network more consistent for the NOC
introduction of our customers to the outsourced NOC's: how to change their attitude
keeping the quality of the NOC's in check over time: means & experiences

What we would like to hear from other participants:

procedures: presentation of the NOC's current procedures & who is responsible for maintaining them
authentication: how do you verify the caller / e-mail sender
reponsibility of your NOC: how far does it go?
good & bad experiences with outsourcing
distributed NOC's (as opposed to a NOC on a central location): pro's & con's

DANTE
Other basic information

Organization (tiers, and which tiers are in-house or outsourced);
Responsibilities (e.g. monitoring, configuring/provisioning, direct or indirect repairs, DNS updates, RIPE updates, POP access and deliveries, reporting);
Hours of operation (office and on-call);
Network type (IP only, hybrid, switched/transmission only, LAN only);
Network services: IPv4/IPv6/multicast IPv4/multicast IPv6/VPNs/QoS/etc

Of most interest to learn from others

Main Monitoring systems and alarm thresholds;
Target repair times;
Number (approx) of ticket mail lists the NOC is on, and approx number of tickets received on these lists
Connected networks: Types of organization and approx numbers for subscribers, peers and upstream IP providers
Core equipment vendor/make, approx numbers;

CARNet
We would like to include in our presentation topics about:

CARNet network, users & Core/Access nodes - how to treat different users and different parts of network
CARNet NOC - organization, tiers, team operating hours, knowledge sharing
NMS - our tool functionalities / developement
prodecudes and documentations - what should be strictly put "on paper" (level of knowledge and procedures) from our experience ...

Topics about what we would like to learn from others are:

NOC key performance indexes - how to measure/to improve ?
Knowledge management and NOC staff training
NMS - tools (functionalities)
NOC organization and procedures: Tier1-N functions, # of persons in shifts and in total, work quality control, peak hours
How to work proactively in NOC (active/passive monitoring)

PIONIER
Five topics PSNC would like to present about NOC:

networks PIONIER (NREN) and POZMAN - technical structure (DWDM, switches, routers) and its influence to NOC structure, organizational structure: Layer 1,2 (NOC) and Layer 3 (IP NOC), employees and their functions
procedures used in our NOC
our TTS - public part, experience, missing functions
NOC tools - tools used in the NOC, missing functions, tools developed and maintained by PSNC
Experiences in joint management application design & deployment with commercial company

Five topics (or questions) that PSNC is interested in knowing how other NOCs handle:

how to minimize users calls (public available FAQ? tools? documentation?)
reports - what? how often?
data and tools integration - how? recommended tools?
serving users calls - how to automate it, distributing calls among different NOC administrators
(D-)DoS - how to detect network attacks

ARNES
ARNES could present:

Pros and contras for dedicated NOC personnel
NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)

ARNES would like to hear from others about:

NOC staff job description
NOC assesment and KPI – measuring NOC performance
NOC to user communication (communication with more or less experienced users, how to keep user contact info up-to-date, personalized web interface for NOC to user communication)
NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)
How many customers do you have? (depends on if you are counting persons, networks, hosts,..)
What kind of agreements do you have with service providers and customers (SLA,..)? Without financial info, but just roughly descriptions of responsibilities.

NORDUnet
Things NORDUnet could present:

Troubleticketing system, experiences from building our own.
Managing user interfaces (using people outside the NOC to aggregate user requests)
Using external community tools (letting the community answer questions instead of the NOC).
NOC organisation (staffing, internal escalations, handover and schedule)
Reaching your network from anywhere (out-of-band, in-band and VPN)

Things NORDUnet would like to hear about from others:

How is knowledge dissemination handled between groups and new NOC employees?
How is monitoring and alarm correlation handled between different systems?
How are new services and users provisioned?
What kind of routines are used in regards to change management (e.g. in planned maintenance)?
What other services than the network does your NOC operate and how do they differentiate from the network service?

GNOC at Indiana University
Five topics to introduce the Global Research NOC at Indiana University:

Global Research NOC Service Desk<http://globalnoc.iu.edu/tour.html> provides 24x7x365 technical call center support, trouble ticket management, and workflow support. The service desk is housed in a state-of-the-art call center in Indiana University's Informatics and Communications Technology Complex (ICTC) on Indiana University's Indianapolis campus. This call center features a fully customizable 30-foot screen used to monitor and troubleshoot the current health of the various networks supported. We provide NOC support for the most advanced research and education networks in the country. http://globalnoc.iu.edu/tour.html
Willingness to assist network customers with other services and requests outside of traditional front-line network operations, such as network redesign planning and utilizing our tool set.
A network knowledgeable tier-one Service Desk that can assess network issues intelligently and can provide engineering with helpful information as they begin troubleshooting.
Internally developed tools such our network database http://globalnoc.iu.edu/grnoc-tool-set.html
International networking efforts

Five topics the GlobalNOC at Indiana University would like to know about other NOC's:

What methodologies and tools do you use to understand complex cross-network relationships and dependencies?
How do you organize and update your internal documentation environment for easy access and search, as well as providing clear and accurate information?
How do you integrate the different groups within your NOC (Service Desk-Tier One, Engineering-Tier Two & Three, Systems Engineering, etc) so each has clear expectations of what each group does, with all working as a unified team?
With R&E networking moving towards having more regional connectedness between smaller networks, RON's, universities, etc, are you planning for a new phase in networking where specific operational differences between NOC's may negatively affect the reliability of the services provided? (for example: different response levels from hardware and fiber vendors, vastly different maintenance windows, NOC's not being 24x7, etc.)
What type of training methodologies do you use to bring new employees up-to-speed quickly so they can start making a valuable contribution to the operation?

SWITCH
The topics SWITCH could tell about:

network, equipment, services, customers
management of fibers - with segments from different providers
staff
outsourced services
work flows
tools
experiences and things we like to improve

The topics SWITCH is interested to know from other NOCs:
The same points like above with special focus on:

incident handling
tools and especially
tools to support the distributed handling of incidents

GRNET
GRNET is keen in presenting one of the topics trailed below:

Efficient administration of many Linux based servers,
Providing VPS service,
Network database (GRNnet-DBII),
Visualisation tools,

On the other hand, we would like to hear how our colleagues tackle the followings:

NOC key performance indexes,
Automated SLA monitoring tools,
Network monitoring tools (may be different than the ones above),
operation of CERT team and security policies

RedIRIS
Five topics to introduce RedIRIS NOC:

Type of network. Equipment and type of lines.
Out of band network.
Trouble ticket system.
SLAs with vendors and providers and type of contact.
Organisation of our NOC.

Five topics RedIRIS would like to know about other NOCs:

Organisation and roles of NOC members. Number of levels. Outsourced vs. staff-in NOC. Pros and cons. Functions. Experience with externalized NOC.
Monitoring tools.
Acquiring, management and operation of dark fibre. Procedures. Optical NOC differenciated from IP NOC?
Services managed by the NOC.
Network database and integration with other monitoring tools.

FCCN
Five topics to introduce FCCN's NOC:

Deployed network (Optical/Ethernet/IP)
Out-of-band systems (MRV using GSM/GPRS)
Service Desk & Automated reporting (per member)
Deployed probes network (QoSMetrics solution)
Past horror stories: What you can try to avoid & Murphy's Law.

Five topics FCCN would like to know about other NOCs:

How is the 24x7x365 NOC service guaranteed?
Which monitoring tools are you currently using. Which ones have you already used in the past and were abandoned. How do you keep network/topology maps updated?
Which services and incidents are covered by the NOC? How do you deal with requests/incidents that are outside your NOC's scope?
Do you have/use a fixed maintenance window?
How do you deal with performance issues/incidents and e2e issues regarding traffic that flows through your network.

CERN
CERN would be glad to present from the following list of topics:

Transforming 2 layer network support into 3 layer support with outsourcing
Integration of network and telecom support
Creation of a knowledge repository for network/telecom support
Structure and roles in a multi-layer NOC operating multi-type networks (campus network, international research network, internet exchange point)
The challenge of using multiple ticketing systems

CERN would be interested in hearing about:

Roles and responsibilities (structure, who does what)
Relation and communication between the network operations team and other internal entities (design, deployment)
How operation is done outside working hours including weekends
How common experience is handled: specialized vs. universal team members
How procedures are created and stored

RESTENA
Five topics to introduce RESTENA's NOC:

Infrastructure building and operations (dark fiber, net devices,server-rooms, POPs, servers)
An "all-in-one" NOC: structure/organization/tools
24x7x365 service support
Coverage of user support (helpdesk)
Services operated (from lambdas to large scale e-mail accounts)

Five topics RESTENA would like to know about other NOCs:

User (customer) information exchange: channels and content
NOC activities/coverage and staff (network, services, community details)
Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and administratively)
Staff organization, i.e. services support strategy (per person/team/NOC) with their respective pros/cons
NOC awareness and identification: internally and among user community

SURFnet
Items we from SURFnet can present our experiences

acquring and managing dark fiber
managing an outsourced NOC
procedures - the good and bad things

Things we would like to hear about from other NOC's:

how to inform/communicate with customers
branding of an outsourced NOC
ticketing systems
CMDB experiences

CESCA
And these are topics that we would like to introduce about our NOC:

Virtualization of the network.
NOC tools and how to adapt them to our needs (development).
Users management. How to approach our users, depending on the action (change, incident, request,..).
Survey for our services, what to ask and how?
Out-of-band access to our network.

These are the topics that we would want to know from other NOCs:

Ticketing Systems, how to optimize their use.
Integration of diferent tools.
Partially outsourced NOCs (24x7, weekends,...): what kind of information do you give them, how do they interact with you and your users?
Escalation of incidents. How many levels?
Documentation: what can and can not be documented.

HEAnet
5 important bullet points introducing our NOC(s)

Heuristic Failure analysis (how we use heuristics to determine SLA calculations and where they are and are not appropriate.) (Also interested in discovering if others are doing this.)
Ticketing, including automatic ticket generation for incidents.
Contract and supplier management - how HEAnet does it and how well it works (or doesn't work.)
Reporting to your clients. How do NRENs report to clients? Periodical and Real-time reporting. On demand reporting.
24 x 7 NOC cover. Do you provide it? What SLA? How do you hire staff and plan rotas, etc? Is there a real client demand for it?

5 questions what you want to know about other NOCs

How are staff hired for the NOC? Skillsets? Interview processes?
Standards compliance and process adoption. Have other NOCs adopted ISO9000? ITIL? PRINCE2? APM? ISO20000? Six Sigma? Other standards?
What scope of services does the NOC support? Networking? Servers? Value added services like DNS, web hosting, etc.? HPC?
How do NOCs source support and SLAs? What about penalty clauses? What's reasonable, what works and how are they enforced?
How do you assign value to what the NOC does? Do you cross charge for projects that will be handed over to the NOC? How to you prove to upper management that the NOC is good value for money?