1、A System Administrators Guide to Preliminary Troubleshooting Techniques,Presented By: Darryl Wright and Volker Kleinschmidt,April 12, 2005,And now a word from our lawyers,Any statements in this presentation about future expectations, plans and prospects for Blackboard and other statements containing
2、 the words “believes,“ “anticipates,“ “plans,“ “expects,“ “will,“ and similar expressions, constitute forward-looking statements within the meaning of The Private Securities Litigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as
3、 a result of various important factors, including the factors discussed in the “Risk Factors“ section of our most recent 10-K filed with the SEC. In addition, the forward-looking statements included in this press release represent the Companys views as of April 11, 2005. The Company anticipates that
4、 subsequent events and developments will cause the Companys views to change. However, while the Company may elect to update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied
5、upon as representing the Companys views as of any date subsequent to April 11, 2005. Blackboard, in its sole discretion, may delay or cancel the release of any product or functionality described in this presentation.,Objectives,Provide Blackboard System Administrators with an overview of their roles
6、, responsibilities, and their impact on the success of the Blackboard Academic Suite at their institutions Highlight best practices for keeping your Blackboard Academic Suite running smoothly and problem free Improve and standardize the troubleshooting process on campus. Methods for better troublesh
7、ooting. Describe ticket submission and escalation process within Blackboard Open Discussion: The evolution of your Blackboard Networked Learning Environment,System Administrator Roles I,Back-end administratorInstallations; webserver admin; database configuration and maintenance; application monitori
8、ng; snapshot and data integration; and even OS maintenance tasks Front-end GUI administratorCourse shell creation; user enrollment; export/import/copy; control over feature availability and design; set policies and business rules for account creation, content preservation etc.; portal administration
9、 System Extension DeveloperCreation of Building Blocks, Application Bridges and Portal Modules using Java and Blackboards APIs,System Administrator Roles II,CMS Help Desk ManagerTroubleshooting issues reported via email, phone, or a ticket system, or self-discovered; escalation of issues to Blackboa
10、rd Technical Support; training and supervision of general help desk staff Trainer creating online documentation and tip sheets Instructional DesignerIn some cases the system administrator may also train instructors and staff on pedagogically sound uses of technology, and participate in the creation
11、of course materials and the tracking and evaluation of student performance,Show of hands,Which roles do you fill at your institution? Back-end administrator Front-end administrator System Extension Developer CMS Helpdesk manager Trainer / Documentation Author Instructional Designer,User community ma
12、iling lists,Back-end administrator Front-end administrator System Extension Developer CMS Helpdesk manager Trainer / Documentation Author Instructional Designer,BBADMIN-L BLKBRD-L BB-OPEN_SRC BB-FS BB-ID,Evolution of System Administration Roles & Practices,MISSION CRITICAL,EXPLORATORY,SUPPORTED,STRA
13、TEGIC,TRANSFORMATIVE: THE NETWORKED LEARNING ENVIRONMENT,Phase I,Phase 2,Phase 3,Phase 4,Phase 5,INSTITUTIONAL GROWTH,Multiple People,Single Server Single Admin,Integrated Production Environment,Monitoring, Backup & Maintenance Practices,Ticketing System,Student /Faculty Training Programs,System Upd
14、ate Deployment Plans,Custom Instructional Tools,Long Range Planning,Help Desk Management,High Performance / High Availability,Strategic Owner,Business Continuity,Identified system manager,Help Desk,Change management practices,Best Practices,Be Prepared Avoiding is better than solving Have a well-est
15、ablished troubleshooting process on campus Train your users to be aware of pitfalls Document local policies and differences Top 10 tips for System Administrators,Best practices - Top 10 tips for System Administrators,Maintain current Operations Workbook Document all server and application changes Ma
16、intain a Test server environment Set up and test a Disaster Recovery Plan Plan ahead for increased usage Train users well to avoid problems Keep it clean - purge old records Manage and monitor your application logs Graph system metrics (MRTG, cricket) Participate in user community,Top 10 Best practi
17、ces for Course Management beginning of semester operational readiness,Fully document your system environment. This documentation should include the systems, applications, as well as the owners. Fill in the Operations Workbook. Practice change management. Make sure all changes to your configuration a
18、re logged, and that a rollback procedure is in place. If something goes wrong the first question should be “what changed?“ Maintain staging and test servers. Ensure that any change to your environment goes through an acceptance testing process. Use the test server to prototype and model changes, and
19、 the staging server for acceptance testing. Create daily full backups and regularly practice a restore to a test server. Understand how many courses and users will be / have been added between semesters. Analyze whether your hardware is powerful enough to handle the increase. Train your users. Sched
20、ule enough workshops early on to highlight new features, and provide actual use cases for better user adoption. Have documentation ready early. Purge records. Use the tools available to archive out courses a pre-determined period after the course is over. Maintain the archive for long term storage a
21、nd record keeping; and then purge the course from your system. Keeping unused courses and accounts on your system can slow performance. Know your logs. Enterprise applications have a number of error and system logs. Monitor them for unusual events and size. Make sure tasks such as log rotation and s
22、tats purging are scheduled appropriately. Verify regularly that they are working correctly. Actively monitor your environment using tools like Big Brother and graph crucial performance data with MRTG, cricket, or similar tools. Participate in the user community. Share experiences, gather resources a
23、nd tips, hear about possible problems before they hit you.,Network diagram (workbook),Use Visio templates or simple flowcharting tool Make sure to label machines with function, hostname and IP-address Indicate ports used on each connection Include servers, firewalls, routers, proxies, load-balancers
24、 Keep it up-to-date,Disaster Recovery Plan,Ready adequate replacement hardware Set up full daily backups to it Test recovery regularly Train multiple staff on recovery Document all necessary steps in standard location (Workbook, Backup/Recovery guide, local customizations) Dont commit replacement sy
25、stem to other production-grade tasks,Sample performance graphs,CPU-busy stats weekly graph,dual weekly graph for total number of logins per day and number of recent logins (last 10 mins),Part II,Troubleshooting,Troubleshooting Process on Campus,The SOAP Method - A Basic Troubleshooting MethodSubject
26、ive - Who?, What?, Where? and When? Objective - Steps to recreate, Error messages, log collection and access informationAssessment - Review steps, errors, logs and assess whether it can be resolved or has to be escalated Plan - Resolution steps or WorkaroundExamples of SOAP interaction:- Used by the
27、 Helpdesk- troubleshooting end-user issues - Used by the System Administrator - recognizing, analyzing, reporting system-issues, focused collection of info for escalation,SOAP at the Helpdesk,User- or course-specific issue? if so, need exact steps and location Browser-specific? test on different mac
28、hines and browsers Recent changes? OS or software update, configuration change, malware infection Restore known-good configuration system restore, spyware removal, virus scan,Troubleshooting User-Reported Issues,Example: Resolving a login problem,Account exists? Correct password used? Caps? If campu
29、s-wide pwd, does it work elsewhere? Correct URL used? Machine-specific? Browser configuration (SSL, JS, cache) Spyware and virus check Create account, reset password, clean machine,SOAP for the GUI admin,Word-of-mouth vs. direct report Issue affects multiple users? Issue repeatable? Issue known in u
30、ser community? Useful info in logs? Report results of this investigation when filing support case,Example: Disappearing forum posts,Group forum or course forum? Who has forum admin rights? Can authors delete own posts? whodunnit: check modperl access logs for POST requests in timeframe in question I
31、nform users that forums are not personal mailboxes and read messages should not be deleted Restore backup to test machine and send text of lost messages to users for re-posting,SOAP for the back-end admin,Intermittent problem vs. event driven Issue affects server performance? Issue repeatable? Issue
32、 tracking in the system logs? Useful info in server OS logs? Report results of this investigation when filing Blackboard support case.,Example: Database Connection Unavailable,Blackboard application not working Users reporting error messages upon connection Problem is not browser specific. Review of
33、 Log files Server OS, DB Logs and Blackboard log files Log results Blackboard log files revealed full transaction logs. Server OS logs reported low drive space on DB server. Database Maintenance Plans System Administrator scheduled Database maintenance routines, Data integrity checks, Backup and tru
34、ncation of logs. Updated Blackboard maintenance routines to include nightly and weekly DB backups,The Escalation Process within Blackboard Client Support,Clarifying the process towards realistic user expectations-sample case lifecycle-sample case interaction between customer and Blackboard support H
35、ow and where can you as the system administrator help to resolve cases quickly and smoothly? - SOAP,Sample case lifecycle,Customer: case creation on Behind the Blackboard TSM: replicates problem TSM: researches KB, Wiki, tracking system for similar cases and known resolution TSM: obtains access info
36、 from customer TSM: creates bug report and escalates case PSE: verifies problem and attempts resolution PSE: escalates to Development Dev, PSE: Analysis of customer impact; prioritization Dev: Developer assigned; target release version determined Dev: Issue is replicated; root cause is determined PS
37、E: communicates between customer and development Dev: solution is created Dev: fix goes to QA Customer: Beta-release testing Dev: Software updated released,Roadblocks in the lifecycle or: why do things take so long?,Communication failures Everyone is too busy Incorrect or incomplete information Cust
38、omer impact not clear Issue is not high priority Other issues hold up a release,Sample case interaction,C: Our system keeps going down S: Symptoms please? Logs? Access? C: Supplies some logs & access info S: Need recent log and perf-report S: Access not working, SSH blocked C: now SSH works S: escal
39、ating S: perf-report useless, no DB info in it you get the picture,Or how it should be,C: occasionally modperl crashes with CPU exhaustion C: logs and perf-report from latest crash, access info S: this has been reported by other customers have you recently updated any building blocks? C: yes, block
40、XYZ S: any courses with course menu links to this BBB? If so, export sample such course, restore to test server, and see whether accessing that course makes modperl CPU usage rise over all bounds C: verified, heres sample course export S: workaround: remove bad menu items in DB,How can YOU help?,Cle
41、ar problem description Precise steps for replication Preliminary analysis (SOAP) Supply requested data promptly Check case status online Focus on important issues We are all only human,Open Discussion,The Evolution of the Networked Learning Environment (John Fontaine, Jeff Kelley, John Knight) Manag
42、ing the growth of your installation of Blackboard Academic Suite Scaling to the Blackboard Enterprise Application Strategy and planning for future growth Blackboard Consultings role in evolution of the NLE Consider Blackboard ASP Services,Evolution of System Administration Roles & Practices,MISSION
43、CRITICAL,EXPLORATORY,SUPPORTED,STRATEGIC,TRANSFORMATIVE: THE NETWORKED LEARNING ENVIRONMENT,Phase I,Phase 2,Phase 3,Phase 4,Phase 5,INSTITUTIONAL GROWTH,Multiple People,Single Server Single Admin,Integrated Production Environment,Monitoring, Backup & Maintenance Practices,Ticketing System,Student /F
44、aculty Training Programs,System Update Deployment Plans,Custom Instructional Tools,Long Range Planning,Help Desk Management,High Performance / High Availability,Strategic Owner,Business Continuity,Identified system manager,Help Desk,Change management practices,Wrapping up,Innovating Together in 05:
45、Use SOAP method for troubleshooting Preparation is better than problem solvingResources Available: Operations Workbook http:/ Training Centerhttp:/ Web Forumshttp:/ up Contact(s): Support site http:/ Product Support - Domestic 1-800-788-5264, International 1-202-715-6019 IF YOU ONLY REMEMBER 1 THING: Know your system well!,Thank You!,
