You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Current »

arXiv-NG Technical Evaluation Rubric v2

RATING KEY: 

no

weak

neutral

good

excels

 

 

 

 

 

  

 

PART 1: ENVIRONMENTAL EVALUATION

Candidate 1

[name]

Candidate 2

[name]

1. Fitness for Purpose - arXivNG

Priority

Rating

Rating

a. Avoids architectural lock-in (one platform; integrated web app)

 

 

 

b. Maximizes potential for strategic evolutionary strategy

 

 

 

c. Minimizes need to re-architect or develop code for arXiv

 

 

 

d. Easily integrated with existing arXiv workflows and services

 

 

 

e. Designed with assumption of distributed system

 

 

 

f. Well-defined APIs for arXiv essential components

 

 

 

g. APIs for arXiv submission, moderation, admin

 

 

 

h. API for arXiv export features

 

 

 

2. Application Architecture

Priority

Rating

Rating

a.Extensible and modular framework

 

 

 

b. Containerization and nodes in cloud

 

 

 

c. Amenable to horizontal scaling

 

 

 

d. Amenable to vertical scaling

 

 

 

3. Code Quality (for existing open source)

Priority

Rating

Rating

a. Well written code

 

 

 

b. Well-defined APIs

 

 

 

c. Open source license

 

 

 

d. Preferred language (Python/Ruby/Java; JavaScript framework)

 

 

 

e. Well supported (regular bug fixes; upgrades; documentation)

 

 

 

f. Low barrier to contribution

 

 

 

g. Best practices in coding: tests, version control, documentation, code guidelines, automated build, automated tests

 

 

 

h. Dependency management

 

 

 

4. Technical Infrastructure

Priority

Rating

Rating

a. Easy to setup a developer environment

 

 

 

b. Deploy and run services in cloud (e.g., VM images + application components; Docker)

 

 

 

c. Elastic provisioning

 

 

 

d. Flexible logging

 

 

 

e. System monitoring (cron; periodic monitoring scripts)

 

 

 

f. Log monitoring and system alerts

 

 

 

g. Security monitoring

 

 

 

h. Service-level agreements

 

 

 

i. Uniform code management

 

 

 

j. Admin access to servers and VMs

 

 

 


 

 

PART 2: FUNCTIONAL EVALUATION

 

Candidate 1

Candidate 2

1. Data model

Priority

Rating

Rating

a. Generic (core data model)

 

 

 

b. Flexible and Extensible (core model can be extended)

 

 

 

c. Model for key entity types (e.g., publication entities; scholarly objects; research objects)

 

 

 

d. Supports relationships (e.g. part/whole; reference links; semantic

 

 

 

e. Supports custom metadata

 

 

 

f. Supports multiple identifiers (local and global)

 

 

 

g. Supports aggregation of entities (e.g., via part/whole relationships)

 

 

 

h. Amenable to serialization in multiple standard formats

 

 

 

2.  Metadata Store

Priority

Rating

Rating

a. Flexible database schema

 

 

 

b. Ability to store metadata as objects (e.g., json)

 

 

 

3.  Object Store

Priority

Rating

Rating

a. API is well-defined and documented

 

 

 

b. Flexible content type (text, binary, data, video, image, PDF, other)

 

 

 

c. CRUD operations on objects upon objects in persistent storage

 

 

 

d. Versioning of objects

 

 

 

e. Backup of objects

 

 

 

f. Redundancy/replication of objects

 

 

 

g. Pluggable storage backend (e.g., file system; cloud storage; storage system)

 

 

 

h. Audit trail of changes

 

 

 

i. Create checksum of objects

 

 

 

j. Monitor object integrity (e.g., via checksums)

 

 

 

4. Workflows

Priority

Rating

Rating

a. Specification of workflows (outside of code)

 

 

 

b. Modern, reactive user interfaces for agents of workflows

 

 

 

    • Submission

 

 

 

    • Moderation

 

 

 

c. Execution and tracking of workflows

 

 

 

    • Submission

 

 

 

    • Moderation

 

 

 

d. Integration with external services (e.g., call outs)

 

 

 

    • Classifier, overlap, TeX,

 

 

 

    • Reference link extraction, metadata extraction

 

 

 

    • iThenticate

 

 

 

e. User communication and notification

 

 

 

    • email

 

 

 

    • Messaging

 

 

 

    • Event audit trails (also visible via user web pages)

 

 

 

f. Logging and history (e.g., of submitters and submissions)

 

 

 

g. RT integration or similar functionality

 

 

 

5. Publishing

Priority

Rating

Rating

a. Persistent URLs for arXiv papers

 

 

 

b. Pre-print DOIs

 

 

 

c. Changes at specific times: freeze time/ publish time

 

 

 

d. Move metadata and content to published storage

 

 

 

e. Triggers for announcements

 

 

 

    • email announcements to submitters

 

 

 

    • email announcements of articles

 

 

 

6. Discovery and Access

Priority

Rating

Rating

a. Indexing easily configured

 

 

 

b. Robust search engine (e.g., Elastic; Solr)

 

 

 

c. Faceted browse

 

 

 

7. Services

Priority

Rating

Rating

a. TeX system - Ability to run as  external service for processing of submission, moderation, publishing of articles.

 

 

 

b. Overlap detection

 

 

 

c. Classification

 

 

 

d. Reference extraction

 

 

 

8. User Interfaces

Priority

Rating

Rating

a. All Web UIs use modern javascript framework

 

 

 

b. Out-of-box moderation UI

 

 

 

c. Custom-built moderation workflow UI

 

 

 

d. Out-of-box submission UI

 

 

 

e. Custom-built submission workflow UI

 

 

 

f. Out-of-box browsing UI and preprint page

 

 

 

g. Custom-built browsing UI and preprint page

 

 

 

h. Out-of-box administrative UI

 

 

 

i. Custom-built administrative UI

 

 

 

j. Faceted search UI

 

 

 

9. User account management

Priority

Rating

Rating

a. Integrating existing user account data and history

 

 

 

b. Flagging user actions

 

 

 

c. Log in as a user for dev purposes

 

 

 

d. ORCID integration

 

 

 

e. Open standards

 

 

 

f. OAuth2

 

 

 

g. User action auditing

 

 

 

10. Email

Priority

Rating

Rating

a. Sending nightly email

 

 

 

b. Totally Decoupled

 

 

 

c. Large scale (~10K daily)

 

 

 

d. Ability to send one-off emails

 

 

 

e. emails to specific users

 

 

 

f. Public relation emails

 

 

 

g. emails to sets of users (such as those affected by an incident)

 

 

 

h. Emails to users via user account system

 

 

 

i. User subscriptions to nightly email categories via email.

 

 

 

11. I18N and accessibility

Priority

Rating

Rating

a. Unicode character handling

 

 

 

b. Should at least not cause problems in this area

 

 

 

12. CMS or Wiki (for public, admin and policy documentation)

Priority

Rating

Rating

a. Public relation features like front page news posting

 

 

 

b. RSS feed not tied to records/articles

 

 

 

c. Help and FAQs pages

 

 

 

d. Admin policies

 

 

 
  • No labels