arXiv-NG Technical Evaluation Rubric v2
RATING KEY:
...
no
...
weak
...
neutral
...
good
...
excels
...
...
...
...
...
...
PART 1: ENVIRONMENTAL EVALUATION
...
Candidate 1
[name]
...
Candidate 2
[name]
...
1. Fitness for Purpose - arXivNG
...
Priority
...
Rating
...
Rating
...
a. Avoids architectural lock-in (one platform; integrated web app)
...
...
...
...
b. Maximizes potential for strategic evolutionary strategy
...
...
...
...
c. Minimizes need to re-architect or develop code for arXiv
...
...
...
...
d. Easily integrated with existing arXiv workflows and services
...
...
...
...
e. Designed with assumption of distributed system
...
...
...
...
f. Well-defined APIs for arXiv essential components
...
...
...
...
g. APIs for arXiv submission, moderation, admin
...
...
...
...
h. API for arXiv export features
...
...
...
...
2. Application Architecture
...
Priority
...
Rating
...
Rating
...
a.Extensible and modular framework
...
...
...
...
b. Containerization and nodes in cloud
...
...
...
...
c. Amenable to horizontal scaling
...
...
...
...
d. Amenable to vertical scaling
...
...
...
...
Priority
...
Rating
...
Rating
...
a. Well written code
...
...
...
...
b. Well-defined APIs
...
...
...
...
c. Open source license
...
...
...
...
d. Preferred language (Python/Ruby/Java; JavaScript framework)
...
...
...
...
e. Well supported (regular bug fixes; upgrades; documentation)
...
...
...
...
f. Low barrier to contribution
...
...
...
...
g. Best practices in coding: tests, version control, documentation, code guidelines, automated build, automated tests
...
...
...
...
h. Dependency management
...
...
...
...
4. Technical Infrastructure
...
Priority
...
Rating
...
Rating
...
a. Easy to setup a developer environment
...
...
...
...
b. Deploy and run services in cloud (e.g., VM images + application components; Docker)
...
...
...
...
c. Elastic provisioning
...
...
...
...
d. Flexible logging
...
...
...
...
e. System monitoring (cron; periodic monitoring scripts)
...
...
...
...
f. Log monitoring and system alerts
...
...
...
...
g. Security monitoring
...
...
...
...
h. Service-level agreements
...
...
...
...
i. Uniform code management
...
...
...
...
j. Admin access to servers and VMs
...
...
...
...
PART 2: FUNCTIONAL EVALUATION
...
Candidate 1
...
Candidate 2
...
1. Data model
...
Priority
...
Rating
...
Rating
...
a. Generic (core data model)
...
...
...
b. Flexible and Extensible (core model can be extended)
...
...
...
...
c. Model for key entity types (e.g., publication entities; scholarly objects; research objects)
...
...
...
...
d. Supports relationships (e.g. part/whole; reference links; semantic
...
...
...
...
e. Supports custom metadata
...
...
...
...
f. Supports multiple identifiers (local and global)
...
...
...
...
g. Supports aggregation of entities (e.g., via part/whole relationships)
...
...
...
...
h. Amenable to serialization in multiple standard formats
...
...
...
...
2. Metadata Store
...
Priority
...
Rating
...
Rating
...
a. Flexible database schema
...
...
...
...
b. Ability to store metadata as objects (e.g., json)
...
...
...
...
3. Object Store
...
Priority
...
Rating
...
Rating
...
a. API is well-defined and documented
...
...
...
...
b. Flexible content type (text, binary, data, video, image, PDF, other)
...
...
...
...
c. CRUD operations on objects upon objects in persistent storage
...
...
...
...
d. Versioning of objects
...
...
...
...
e. Backup of objects
...
...
...
...
f. Redundancy/replication of objects
...
...
...
...
g. Pluggable storage backend (e.g., file system; cloud storage; storage system)
...
...
...
...
h. Audit trail of changes
...
...
...
...
i. Create checksum of objects
...
...
...
...
j. Monitor object integrity (e.g., via checksums)
...
...
...
...
4. Workflows
...
Priority
...
Rating
...
Rating
...
a. Specification of workflows (outside of code)
...
...
...
...
b. Modern, reactive user interfaces for agents of workflows
...
...
...
...
- Submission
...
...
...
...
- Moderation
...
...
...
...
c. Execution and tracking of workflows
...
...
...
...
- Submission
...
...
...
...
- Moderation
...
...
...
...
d. Integration with external services (e.g., call outs)
...
...
...
...
- Classifier, overlap, TeX,
...
...
...
...
- Reference link extraction, metadata extraction
...
...
...
...
- iThenticate
...
...
...
...
e. User communication and notification
...
...
...
...
...
...
...
...
- Messaging
...
...
...
...
- Event audit trails (also visible via user web pages)
...
...
...
...
f. Logging and history (e.g., of submitters and submissions)
...
...
...
...
g. RT integration or similar functionality
...
...
...
...
5. Publishing
...
Priority
...
Rating
...
Rating
...
a. Persistent URLs for arXiv papers
...
...
...
...
b. Pre-print DOIs
...
...
...
...
c. Changes at specific times: freeze time/ publish time
...
...
...
...
d. Move metadata and content to published storage
...
...
...
...
e. Triggers for announcements
...
...
...
...
- email announcements to submitters
...
...
...
...
- email announcements of articles
...
...
...
...
6. Discovery and Access
...
Priority
...
Rating
...
Rating
...
a. Indexing easily configured
...
...
...
...
b. Robust search engine (e.g., Elastic; Solr)
...
...
...
...
c. Faceted browse
...
...
...
...
7. Services
...
Priority
...
Rating
...
Rating
...
a. TeX system - Ability to run as external service for processing of submission, moderation, publishing of articles.
...
...
...
...
b. Overlap detection
...
...
...
...
c. Classification
...
...
...
...
d. Reference extraction
...
...
...
...
8. User Interfaces
...
Priority
...
Rating
...
Rating
...
a. All Web UIs use modern javascript framework
...
...
...
...
b. Out-of-box moderation UI
...
...
...
...
c. Custom-built moderation workflow UI
...
...
...
...
d. Out-of-box submission UI
...
...
...
...
e. Custom-built submission workflow UI
...
...
...
...
f. Out-of-box browsing UI and preprint page
...
...
...
...
g. Custom-built browsing UI and preprint page
...
...
...
...
h. Out-of-box administrative UI
...
...
...
...
i. Custom-built administrative UI
...
...
...
...
j. Faceted search UI
...
...
...
...
9. User account management
...
Priority
...
Rating
...
Rating
...
a. Integrating existing user account data and history
...
...
...
...
b. Flagging user actions
...
...
...
...
c. Log in as a user for dev purposes
...
...
...
...
d. ORCID integration
...
...
...
...
e. Open standards
...
...
...
...
f. OAuth2
...
...
...
...
g. User action auditing
...
...
...
...
10. Email
...
Priority
...
Rating
...
Rating
...
a. Sending nightly email
...
...
...
...
b. Totally Decoupled
...
...
...
...
c. Large scale (~10K daily)
...
...
...
...
d. Ability to send one-off emails
...
...
...
...
e. emails to specific users
...
...
...
...
f. Public relation emails
...
...
...
...
g. emails to sets of users (such as those affected by an incident)
...
...
...
...
h. Emails to users via user account system
...
...
...
...
i. User subscriptions to nightly email categories via email.
...
...
...
...
11. I18N and accessibility
...
Priority
...
Rating
...
Rating
...
a. Unicode character handling
...
...
...
...
b. Should at least not cause problems in this area
...
...
...
...
...
...
...
...
12. CMS or Wiki (for public, admin and policy documentation)
...
Priority
...
Rating
...
Rating
...
a. Public relation features like front page news posting
...
...
...
...
b. RSS feed not tied to records/articles
...
...
...
...
c. Help and FAQs pages
...
...
...
...
d. Admin policies
...
...
...
arXiv-NG Technical Evaluation Rubric v2
...