The content on this wiki page has been moved to https://confluence.cornell.edu/x/pLBRF.This page is no longer kept up to date. |
Oya Y. Rieger, arXiv Program Director, Cornell University Library, June 2016
Acknowledgements: Many individuals were involved in designing and testing the survey and helped out with the data analysis. Special thanks go to Deborah Cooper, Andrea Cruz, Jim Entwood, Martin Lessmeister, Leah McEwen, Chloe McLaren, Chris Myers, David Ruddy, Vandana Shah, Gail Steinhart, Simeon Warner, and Jake Weiskoff. Also, we are grateful for the guidance from the arXiv’s Member Advisory Board and Scientific Advisory Board.
EXECUTIVE SUMMARY
As part of its 25th anniversary vision-setting process, the arXiv team at Cornell University Library conducted a user survey in April 2016 to seek input from the global user community about the current services and future directions. We were heartened to receive 36,000 responses, representing arXiv’s diverse community (See Appendix A). The prevailing message is that users are happy with the service as it currently stands. 95% of survey respondents said that they are very satisfied or satisfied with arXiv. Furthermore, 72% of respondents indicated that arXiv should focus on its main purpose, which is to quickly make available scientific papers, and this will be enough to sustain the value of arXiv in the future. This theme was pervasively reflected in the open text comments. A significant number of respondents suggested keeping to the core mission and enabling arXiv’s partners and related service providers to continue to build new services and innovations on top of arXiv.
Many of the comments reflected deep satisfaction with and gratitude for arXiv. Several users referred to the significance of the service for their personal career development and expressed thanks for its continued existence; for example, a typical comment was: “Thanks for the hard work of many people over the years. My work life would be very different without your efforts.” arXiv also received many plaudits for advancing the dissemination of research through the open-access system. One user referred to the service as “a beacon for scientific communication.” Several commenters expressed how crucial arXiv has been for them personally in enabling them to quickly access the latest research in their field. There was an overall perception that arXiv was an important leader in the development of alternatives to traditional publishing. Independent researchers who are unaffiliated with large institutions and who might otherwise have delayed access to papers particularly emphasized the importance of arXiv for their work.
The combination of multiple choice responses (see Appendix B) and the extensive and thoughtful open text comments pinpointed areas that need to be upgraded and enhanced. Improving the search function emerged as a top priority as the users expressed a great deal of frustration with the limited search capabilities currently available, especially in author searches. Providing better support for submitting and linking research data, code, slides and other materials associated with papers emerged as another important service to expand. Regardless of their subject area, users were in agreement about the importance of continuing to implement quality control measures, such as checking for text overlap, correct classification of submissions, rejection of papers without much scientific value, and asking authors to fix format-related problems. Several users commented on the need to randomize the order of new papers in announcements and mailings. There were several useful remarks about the need to improve the endorsement system and provide more information about the moderation process and policies.
In regard to arXiv’s role in scientific publishing, some users encouraged the arXiv team to think boldly and further advance open access (and new forms of publishing) by adding features such as peer review and encouraging overlay journals. On the other hand, many users strongly emphasized the importance of sticking to the main mission and not getting side-tracked into formal publishing. There was a similar divergence of opinion about encouraging an open review process by adding rating and annotation features. When it comes to adding new features to arXiv to facilitate open science, the prevailing opinion was that any such features need to be implemented very carefully and systematically, and without jeopardizing arXiv’s core values.
While many respondents took the time to suggest future enhancements or the finessing of current services, several users were strident in their opposition to any changes. Throughout all of the suggestions and regardless of the topic, commenters unanimously urged vigilance when approaching any changes and cautioned against turning arXiv into a “social media” style platform. The feeling is that arXiv as it exists is working well and while there are some areas for improvement, too much change could potentially weaken the effectiveness and overall mission of arXiv.
KEY FINDINGS
Improving the Current arXiv Services
Importance of Quality Control Measures
Adding New Subject Categories:
Developing New Services
Finding arXiv Papers:
Also see:
http://arxiv.org/abs/1607.08212
About arXiv: arXiv, an open-access scientific digital archive, is funded by the Simons Foundation, Cornell University Library, and about 190 member libraries from all around the world. The site is collaboratively governed and supported by the research communities and institutions that benefit from it most directly, ensuring a transparent and sustainable resource. It is a moderated scholarly communication forum informed and guided by scientists and the scientific cultures it serves. As of June 2016, arXiv contains more than 1,110,000 e-prints. In 2015, the repository saw 105,000 new submissions and close to 139 million downloads from all over the world.
I use arXiv in the following ways: (Please choose all that apply)
Answer | % | Count |
I am an arXiv reader | 93% | 31862 |
I am an arXiv author | 53% | 18270 |
I am an arXiv submitter | 50% | 17189 |
I am an arXiv (other type of user): Please describe | 2% | 845 |
The number of articles I have published/submitted on arXiv is:
Answer | % | Count |
1 article | 11.99% | 2570 |
2 articles | 8.96% | 1920 |
3 - 4 articles | 15.19% | 3254 |
5-10 articles | 23.06% | 4941 |
More than 10 articles | 40.80% | 8743 |
Total | 100% | 21428 |
My current occupation is: (Please choose ALL that apply)
Answer | % | Count |
I am an academic faculty member (professor) at a college or university | 27% | 8868 |
I am an academic staff member (researcher or postdoc) at a college or university | 22% | 7207 |
I am a researcher at a non-profit or governmental agency | 8% | 2707 |
I am a Masters/Ph.D. student | 30% | 9890 |
I am an undergraduate student | 5% | 1514 |
I am (please describe) | 13% | 4353 |
13% of respondents (4353) indicated a different occupation category. The top ones included researchers at a company or industry (900), engineer (515), and retired individuals (478). There were also respondents who described themselves as science writers, editors, or freelance editors. Other response types included data scientist, self-described amateur researchers, self-described laypeople, unemployed, teachers, and the generally curious (e.g., “a man doing research as hobby”).
As a user, my main subject area of interest in arXiv is: (please choose all that apply)
Almost 2,000 respondents checked the Other option to specify their main area of interest. The top categories were astrophysics (726) and astronomy (653).
I have been using arXiv for:
Answer | % | Count |
0 - 2 years | 19.54% | 6470 |
3 - 5 years | 28.96% | 9592 |
6- 10 years | 25.44% | 8425 |
11 or more years | 26.06% | 8632 |
Total | 100% | 33119 |
My age is:
Answer | % | Count |
younger than 30 years | 37.42% | 12364 |
30 - 39 years | 31.27% | 10332 |
40 - 49 years | 13.76% | 4545 |
50 - 59 years | 9.30% | 3073 |
60 - 69 years | 5.77% | 1908 |
70 years and over | 2.47% | 817 |
Total | 100% | 33039 |
Q6 - My main place of work is located in:
Other Countries: 1% or less representation each from 113 countries
How important is it to improve on the following CURRENT arXiv services?
Question | Very important | Somewhat important | Not important & should not be doing this | No |
Improve search functions to allow more refined results (e.g., narrow down results by additional search terms, filter by publication year or institutional affiliation, etc.): | 70.38% | 19.34% | 6.14% | 4.13% |
Improve support for submitting research data, code, slides, and other materials associated with a paper (e.g., I want to be able to upload my datasets/machine- readable tables with my article): | 41.95% | 22.64% | 14.03% | 21.37% |
Improve support for linking research data, code, slides, and other materials associated with a paper (e.g., I want to be able to link to my slides on SlideShare): | 40.65% | 25.20% | 17.70% | 16.45% |
Improve support for submitting research papers by updating the TeX engine: | 39.36% | 23.17% | 16.71% | 20.76% |
Improve the email alert system so that readers can customize their settings and choose to receive alerts about specific sub-topics: | 37.85% | 26.48% | 20.25% | 15.42% |
Improve the trackback mechanism (linking papers back to blogs and commentaries that cite thos papers): | 36.52% | 29.50% | 20.30% | 13.67% |
Simplify the submission process by providing clearer instructions and simpler language: | 32.45% | 22.55% | 25.20% | 19.80% |
How important is it to develop the following NEW arXiv services? -
Question | Very important | Somewhat important | Not important & should not be doing this | No opinion |
Add direct links to papers in the references (support reference extraction): | 63.04% | 26.89% | 5.78% | 4.29% |
Offer citation export in formats such as BibTeX, RIS, etc.: | 57.68% | 23% | 10.95% | 8.37% |
Enable extraction for the BibTeX entry for the arXiv citation: | 55.54% | 23.82% | 9.72% | 10.91% |
Provide Citation Analysis tools (examining the frequency and pattern of a paper's citation): | 52.95% | 27.08% | 14.28% | 5.69% |
Support compliance with public/open access mandates (funding agency policies that require research results to be made public) by allowing final versions of papers to be submitted with information such as funding sources and grant numbers: | 42.06% | 26.21% | 13.68% | 18.05% |
Enable submitting an article to a journal at the same time as it is uploaded to arXiv: | 39.28% | 23.09% | 25.23% | 12.40% |
Offer a rating system so readers can recommend arXiv papers that they find valuable: | 36.28% | 21.76% | 35.56% | 6.40% |
Enable linkages (interoperability) with other repositories (e.g., run by libraries), so that a paper accepted by arXiv is accepted at the same time by the other repositories: | 35.25% | 28.14% | 17.25% | 19.36% |
Develop an annotation feature which will allow readers to comment on papers: | 34.89% | 23.62% | 34.08% | 7.41% |
Where do you go to find arXiv papers? Please choose all that apply:
Answer | % | Count |
Go directly to arXiv.org (arXiv homepage) | 79% | 22804 |
ADS | 14% | 4144 |
Inspire | 13% | 3773 |
Google Scholar | 35% | 10016 |
Google search engine | 50% | 14440 |
arXiv email alerts | 14% | 4086 |
Other search engines | 5% | 1402 |
Subject gateways for arXiv, such as the Math Front | 4% | 1203 |
Other (please specify): | 9% | 2662 |
If you have used the arXiv homepage for finding papers, how easy is it to navigate?
Answer | % | Count |
Very easy | 14.85% | 3916 |
Easy | 32.05% | 8450 |
Somewhat easy | 25.20% | 6644 |
Somewhat difficult | 21.60% | 5696 |
Difficult | 5.02% | 1324 |
Very difficult | 1.27% | 336 |
Total | 100% | 26366 |
If you have used the arXiv homepage, how do you usually navigate our main page? Please choose all that apply.
Answer | % | Count |
Go to link "new" or "recent" under a particular category | 63% | 16503 |
Use arXiv search engine and enter a specific arXiv-id, author name, or search term | 63% | 16478 |
Receive daily mailing list, and then look for a particular article on the search field | 14% | 3692 |
Other, please explain: | 3% | 853 |
How important are the following CURRENT quality control measures?
Question | Very important | Somewhat important | Not important & should not be doing this | No opinion |
arXiv checks papers for text overlap: an author's use of too much identical text from other authors' papers, without making it clear that the text is not their own material, i.e., "plagiarism": | 77.41% | 14.66% | 4.96% | 2.96% |
arXiv makes sure submissions are correctly classified (the subject categories are included on the arXiv homepage): | 64.38% | 25.32% | 7.01% | 3.29% |
arXiv keeps out (rejects) papers that don't have much scientific value: | 60.02% | 19.14% | 15.49% | 5.35% |
arXiv checks papers for too much text re-use from an author's earlier works, i.e., "self-plagiarism" (reuse of identical content from one's own published work without citing): | 57.77% | 24.64% | 14.08% | 3.51% |
arXiv checks papers for format-related problems (line numbers in text, missing references, oversize submissions, etc.) and asks authors to fix them before they are announced:. | 55.00% | 29.83% | 11.51% | 3.66% |
arXiv moderates the scientific content of trackback (links to blogs and commentaries) before permitting the link to be added: | 39.60% | 26.31% | 17.59% | 16.50% |
Please choose any ONE of the following statements that you agree with the most:
Overall, how satisfied are you with arXiv?
Answer | % | Count |
Very satisfied | 52.92% | 14770 |
Satisfied | 42.43% | 11841 |
Somewhat satisfied | 3.55% | 990 |
Somewhat dissatisfied | 0.54% | 150 |
Very dissatisfied | 0.15% | 42 |
No opinion | 0.42% | 116 |
Total | 100% | 27909 |
Which of the following BEST describes your opinion of how arXiv needs to move forward?
Answer | % | Count |
arXiv should focus on its main purpose, which is to quickly make available scientific papers. This will be enough to hold up the value of arXiv in the future. | 71.94% | 19865 |
arXiv should expand its main mission, and spend more time and resources to provide new services. This is necessary to hold up the value of arXiv in the future. | 19.59% | 5410 |
No opinion | 8.47% | 2340 |
Total | 100% | 27615 |