ARL proposed new measures and its potential impact on CUL
May 2, 2007 lm xl

[There is a link to background documents at the end of this report.]

Overall, these new measures add a dimension of output/outcome to ARL's traditional and largely input-focused measures.   The implementation is at a very preliminary stage.  A progress report is expected at the May 2007 ARL Business Meeting.  No definitions have been published.  Thus, the remarks on impact for CUL are rather speculative. 

Background
Two commissioned reports by Yvonna Lincoln and Bruce Thompson are foundational for the ARL Board approved action items.  Many factors that would form the definitions in the future may come from the findings reported in these 2 papers.  Lincoln's paper described:
 
Characteristics of the library of the future:

  • Shift from a teaching-centered vision of universities toward a learning-centered campus.
  • Higher level of agility in collection... Acquisition decision will depend largely on resource sharing arrangement, avoid duplications, and enhance the uniqueness of a library. 
  • New services will come in the form of new relationships with research and teaching personnel, designing new forms of updated technology... new policies.
  • Engage in external fund raising.
  • Develop new skills.
  • Contribution to faculty productivity and competitiveness in funding arenas.

 Critical new metrics for library of the future:

  • Value of consortial arrangement
  • Contribution to quality teaching, to student outcomes (retention and learning), to research productivity etc.
  • What collection is or is not used, is most used, is most intensely used, and the least used.
  • Community-wise, who is and who is not being served.

Translating into measures ARL to take lead on

  • Collection:
    o       Unique aspects.  ARL measures only focus on textual forms, not recognize measure non-textual, such as objects.
    o       Stewardship function, e.g. creating access where non exist before
    o       Critical cultural role, e.g. preservation/digitization.
  • Define consortia, incl. shared resources, shared storage, and other cooperative arrangements, value-adds to participants.
  • Administrative and budgetary efficiencies: efficiency and cost-effectiveness in the context of services rendered; flexibility of fiscal resources, shifts in staff skills in hiring and professional development.
  • Student outcomes/student learning/graduate success: relationship of the library and its collections to undergraduate success; contribution to extending diversity on campus, retention, learning outcome; where grads are hired; what they believe they take with them to jobs etc.
  • Contribution to faculty productivity: curricular design, materials identification and preparation, learning assessment, ongoing work as a part of the research team in data and information management etc.
  • Social frameworks/intellectual networks: cease thinking "library as place" and begin to think in terms of "place as library", i.e., neutral agency, domain agnostic, trans-disciplinary, libraries become intellectual brokers.
  • Generating new knowledge
    o       Navigation of increasingly complex bodies of knowledge and systems of meta-data
    o       Creation of knowledge trees and meta-data organization  system
  • Creating the collective good with reusable assets: the cultural role of libraries.
  • Must consider complications of copyright in many new service forms.

Implementation-        

  • Collect descriptive and narrative data for 3-5 years
  • Aggregate and analyze for common denominators
  • Creation of commonly understood metrics. 

ARL Board approved action items 

It is imperative to understand that ARL's ranking indicates "similarity of size", not quality.  However, ranking has been misperceived for quality often.  CUL needs to take this misperception into consideration when making decisions.  

1. Reserve use of the current membership criteria index for those occasions when it is needed for consideration of membership issues.
      Existing criteria that measure input rather than output/outcome.  For 2004/05, Cornell ranked 9th.

2. Implement an expenditure-focused index.
     CUL has been reporting 4 categories of this data.  Depending on ARL's definitions we project that CUL's ranking won't change drastically.  CUL's 2004/05 rankings are:
        - Total expenditures: 9th
        - Total materials expenditure: 11th
        - FTE professional and support staff combined: 9th
        - Professional staffs' salaries and wages: 16th
        - "professional staff FTE": 24th   
       - Support staff FTE: 5th
       - E-materials expenditure: 13th
     Immediate decision needed: whether to include CUL open lines in the data, whether to include some non-academic exempt staff in our "professional staff" count.

3. Use the new expenditure-focused index for any public reports, such as in the Chronicle of Higher Education.
     Impact on CUL: Unknown.  CUL needs to decide within context. 

4. Experiment with another alternative index that combines the following three factors: collections, services, and collaborative relationships.
     CUL has been reporting data in 9 categories.  CUL's 2004/05 rankings were:

       - Volumes in library: 11th
       - Volumes added (gross): 14th
       - Total FTE non-student staff: 9th  - Linda, what does this one measure here?
       - Total current serials: 6th
       - Participants in group presentations: 20th
       - Library presentations to groups: 15th
       - Reference transactions: 39th
       - Total items borrowed (ILL): 38th
       - Total items loaned (ILL): 14th
     Absent of a definition, it is hard to project the impact on CUL.  Since this is being considered as "another alternative index", how CUL will fare is unknown. 

5. Revise definitions for collections-related data categories currently collected and experiment with a variety of new measures, including usage data, strength of collections, and service quality measures to develop a richer set of variables for potential inclusion in the three-factor alternative index (see above).
     Many of these measures are new, so it's not clear yet how they will affect our standings.  The data collection would require significant resources, new workflow, new and wider collaboration.  Potential data elements for this category are:
       - unique/rare volumes held
       - web based reference transactions
       - shared storage square feet
       - collaboratively-held common holdings
       - Service quality improvement (number of user focus groups conducted, FTE staff assigned to service quality assessment activities, number of continuing education training sessions on service quality improvement attended by library staff)
       - Providing users access to digital content (volume counts, expenditures on digital resources, usage statistics for electronic resources)
     Immediately this year, CUL needs to collect several new expenditure measures that are broken down to finer details that will be difficult, if possible to collect.  Our serial count may drop and this will increase our cost per title/subscription. 
     Immediate decision needed: Ask ARL to confirm that title count replaces the subscription count this coming year.  CUL may need to note its concern that ARL requests for e-expenditure data keep getting more compartmentalized by type of resource while distinctions between formats and billing methods for them keep getting more blurred.
 

6. Collect qualitative data to develop a profile of ARL member libraries.
     This is the most challenging category for CUL because we do not collect many of the data.  We currently do not have a workflow that supports such large-scale, multi-partner data collection.  We will not be able to attain some data since we do not have access to them.  Even with collaboration, we may not attain certain data elements, because, by nature, they are hard to define.  CUL needs to think carefully about the value of the profile, and invest the right resources to develop this profile.  CUL needs to understand how ARL plans to manage across-institutional consistency which has been an issue with the existing data.  Potential, 3 key areas that will form the profile of a future research libraries:
        i.      holdings
        ii.      user interactions
        iii.      collaborations

  • Agility in purchasing unique collections
  • More proactive in developing new services, incorporating new technologies
  • Shifts in skills of staff to be hired, e.g. ability to work across disciplines, grant and contract preparation, marketing, assessment
  • Contributions to retention of undergraduates
  • Post graduate information on students
  • Creating social frameworks and effecting/fostering intellectual networks in ways never undertaken before
  • House cooperative learning projects especially if redesigning.
  • Removal of text/print to storage - replaced by desktop delivery
     

Overall impact on CUL:

  • No immediate, significant impact other than adjustments of data in some sub-categories.
  • If implemented, significant, additional investment in data collection is required; new and significantly wider collaboration with, e.g., CIT and other University units, consortial partners, is a must.
  • Some data may not be practically attainable.
     

Recommendations:

  • Actively participate in ARL decision making process; evaluate and help shape definitions.
  • Discuss/Determine in executive committees what decisions CUL must make for functional areas.
  • Decide what success indicators are, what data captures the success levels, how to organize the data so it reveals the performance.
  • Decide to what extent, if at all, to invest resources for this purpose.
  • Should the new measures require more investment than CUL is prepared to put in, decide on an alternative or even an exit plan.

ARL visiting program officers' findings:

  • Limited resources for assessment
  • Prioritization and ownership of assessment activities
  • Decentralization of the staff
  • Lack of oversight committees for some functional areas
  • Lack of assessment related skills
  • Initiatives tend not to be assessed
  • The challenge of deciding when the benefits of assessment are worth the cost
  • The move from assessment to action can often be stymied
  • Success in usability testing needs to be operationalized
  • Need to utilize measures of success
  • Need a place to store data that people have gathered

Background Documents

  1. TASK FORCE ON NEW WAYS OF MEASURING COLLECTIONS FINAL REPORT (http://www.arl.org/bm~doc/tf_013107_final.pdf)
  2. Yvonna Lincoln & Ruth Harrington. Research libraries as knowledge producers: a shifting context for policy and funding, final technical report to the Task Force on New Ways of Measuring Collections, 2006.
  3. Bruce Thompson. Some alternative quantitative library activity, descriptions/statistics that supplement the ARL logarithmic index.
  • No labels