...
Acknowledgements: Many individuals were involved in designing and testing the survey and helped out with the data analysis. Special thanks go to Deborah Cooper, Andrea Cruz, Jim Entwood, Martin Lessmeister, Leah McEwen, Chloe McLaren, Chris Myers, David Ruddy, Vandana Shah, Gail Steinhart, Simeon Warner, and Jake Weiskoff. Also, we are grateful for the guidance from the arXiv’s Member Advisory Board and Scientific Advisory Board.
...
The combination of multiple choice responses (see Appendix B) and the extensive and thoughtful open text comments pinpointed areas that need to be upgraded and enhanced. Improving the search function emerged as a top priority as the users expressed a great deal of frustration with the limited search capabilities currently available, especially in author searches. Providing better support for submitting and linking research data, code, slides and other materials associated with papers emerged as another important service to expand. Regardless of their subject area, users were in agreement about the importance of continuing to implement quality control measures, such as checking for text overlap, correct classification of submissions, rejection of papers without much scientific value, and asking authors to fix format-related problems. Several users commented on the need to randomize the order of new papers in announcements and mailings. There were several useful remarks about the need to improve the endorsement system and provide more information about the moderation process and policies.
In regard to arXiv’s role in scientific publishing, some users encouraged the arXiv team to think boldly and further advance open access (and new forms of publishing) by adding features such as peer review and encouraging overlay journals. On the other hand, many users strongly emphasized the importance of sticking to the main mission and not getting side-tracked into formal publishing. There was a similar divergence of opinion about encouraging an open review process by adding rating and annotation features. When it comes to adding new features to arXiv to facilitate open science, the prevailing opinion was that any such features need to be implemented very carefully and systematically, and without jeopardizing arXiv’s core values. Link to principles?
While many respondents took the time to suggest future enhancements or the finessing of current services, a distinct group of several users were strident in their opposition to any changes. Throughout all of the suggestions and regardless of the topic, commenters unanimously urged vigilance when approaching any changes and cautioned against turning arXiv into a “social media” style platform. The feeling is that arXiv as it exists is working well and while there are some areas for improvement, too much change could potentially weaken the effectiveness and overall mission of arXiv.
...
- The vast majority of arXiv’s users access the papers directly from the homepage (79%), followed by using Google to search (50%) and Google Scholar (35%).
- Once on the homepage, reactions were mixed regarding the ease of use and navigation. 32% rated this as easy, but only 25% find it somewhat easy and 21.6% rated it somewhat difficult to use.
- To discover content, 63% of users go to the link for new or recent under a particular category and equally 63% of users use arXiv’s search engine and enter a specific arXiv ID, author name or search term. A small number of users, 14%, rely on the daily mailing list and then look for a particular article in the search field.
- In the open text comments, opinion was divided about the user interface. The majority of respondents disliked the outdated style, but a definite subgroup appreciated the interface’s simplicity, which these users feel helps arXiv efficiently carry out its mission. The main issues mentioned aside from the homepage’s look were the number of links, layout and finding submission information. The lack of hierarchy in organization was found challenging to understanding arXiv’s navigation.
- Requests for enhancements related to UX included greater personalization of arXiv for readers; for example, the ability to “favorite” papers, curate a personal library, and see recommendations when users visit the site. Other users mentioned the development of APIs to further facilitate the development of overlay journals. Some users also suggested the development of a mobile-friendly version.
- Many commenters either described how they rely on other services to interact with arXiv content (site-specific searches, ADS, INSPIRE) or recommended features based on their experience with other information systems. Among those frequently praised were ADS, INSPIRE, Google Scholar, gitxiv.com and arxiv-sanity.com.
APPENDIX A: DEMOGRAPHICS OF RESPONDENTS
I use arXiv in the following ways: (Please choose all that apply)
About arXiv: arXiv, an open-access scientific digital archive, is funded by the Simons Foundation, Cornell University Library, and about 190 member libraries from all around the world. The site is collaboratively governed and supported by the research communities and institutions that benefit from it most directly, ensuring a transparent and sustainable resource. It is a moderated scholarly communication forum informed and guided by scientists and the scientific cultures it serves. As of June 2016, arXiv contains more than 1,110,000 e-prints. In 2015, the repository saw 105,000 new submissions and close to 139 million downloads from all over the world.
APPENDIX A: DEMOGRAPHICS OF RESPONDENTS
I use arXiv in the following ways: (Please choose all that apply)
Answer | % | Count |
I am an arXiv reader | 93% | 31862 |
I am an | ||
Answer | % | Count |
I am an arXiv reader | 93% | 31862 |
I am an arXiv author | 53% | 18270 |
I am an arXiv submitter | 50% | 17189 |
I am an arXiv (other type of user): Please describe | 2% | 845 |
...
Q6 - My main place of work is located in:
Total 31,255 responses
Other Countries: 1% or less representation each from 113 countries
APPENDIX B: OPINIONS ON ARXIV'S CURRENT SERVICES & FUTURE DIRECTIONS
How important is it to improve on the following CURRENT arXiv services?
...
Question | Very important | Somewhat important | Not important & should not be doing this | No |
Improve search functions to allow more refined results (e.g., narrow down results by additional search terms, filter by publication year or institutional affiliation, etc.): | 70.38% | 19.34% | 6.14% | 4.13% |
Improve support for submitting research data, code, slides, and other materials associated with a paper (e.g., I want to be able to upload my datasets/machine- readable tables with my article): | 41.95% | 22.64% | 14.03% | 21.37% |
Improve support for linking research data, code, slides, and other materials associated with a paper (e.g., I want to be able to link to my slides on SlideShare): | 40.65% | 25.20% | 17.70% | 16.45% |
Improve support for submitting research papers by updating the TeX engine: | 39.36% | 23.17% | 16.71% | 20.76% |
Improve the email alert system so that readers can customize their settings and choose to receive alerts about specific sub-topics: | 37.85% | 26.48% | 20.25% | 15.42% |
Improve the trackback mechanism (linking papers back to blogs and commentaries that cite thos papers): | 36.52% | 29.50% | 20.30% | 13.67% |
Simplify the submission process by providing clearer instructions and simpler language: | 32.45% | 22.55% | 25.20% | 19.80% |
Number of responses: 26,420
How important is it to develop the following NEW arXiv services? -
Question | Very important | Somewhat important | Not important & should not be doing this | No opinion |
Add direct links to papers in the references (support reference extraction): | 63.04% | 26.89% | 5.78% | 4.29% |
Offer citation export in formats such as BibTeX, RIS, etc.: | 57.68% | 23% | 10.95% | 8.37% |
Enable extraction for the BibTeX entry for the arXiv citation: | 55.54% | 23.82% | 9.72% | 10.91% |
Provide Citation Analysis tools (examining the frequency and pattern of a paper's citation): | 52.95% | 27.08% | 14.28% | 5.69% |
Support compliance with public/open access mandates (funding agency policies that require research results to be made public) by allowing final versions of papers to be submitted with information such as funding sources and grant numbers: | 42.06% | 26.21% | 13.68% | 18.05% |
Enable submitting an article to a journal at the same time as it is uploaded to arXiv: | 39.28% | 23.09% | 25.23% | 12.40% |
Offer a rating system so readers can recommend arXiv papers that they find valuable: | 36.28% | 21.76% | 35.56% | 6.40% |
Enable linkages (interoperability) with other repositories (e.g., run by libraries), so that a paper accepted by arXiv is accepted at the same time by the other repositories: | 35.25% | 28.14% | 17.25% | 19.36% |
Develop an annotation feature which will allow readers to comment on papers: | 34.89% | 23.62% | 34.08% | 7.41% |
Number of responses: 26,890
Where do you go to find arXiv papers? Please choose all that apply:
...
Question | Very important | Somewhat important | Not important & should not be doing this | No opinion |
arXiv checks papers for text overlap: an author's use of too much identical text from other authors' papers, without making it clear that the text is not their own material, i.e., "plagiarism": | 77.41% | 14.66% | 4.96% | 2.96% |
arXiv makes sure submissions are correctly classified (the subject categories are included on the arXiv homepage): | 64.38% | 25.32% | 7.01% | 3.29% |
arXiv keeps out (rejects) papers that don't have much scientific value: | 60.02% | 19.14% | 15.49% | 5.35% |
arXiv checks papers for too much text re-use from an author's earlier works, i.e., "self-plagiarism" (reuse of identical content from one's own published work without citing): | 57.77% | 24.64% | 14.08% | 3.51% |
arXiv checks papers for format-related problems (line numbers in text, missing references, oversize submissions, etc.) and asks authors to fix them before they are announced:. | 55.00% | 29.83% | 11.51% | 3.66% |
arXiv moderates the scientific content of trackback (links to blogs and commentaries) before permitting the link to be added: | 39.60% | 26.31% | 17.59% | 16.50% |
Number of responses: 26,430
Overall, how satisfied are you with arXiv?
...