...
These are files that were directly used for creating a list of simplifications.
It is data aggregated over the original Hadoop processing.
Every file contains sentences that were aligned between some adjacent revisions with a threshold greater
than .3 using a TF-IDF style alignment and where the source phrase and the final phrase were both less than or equal to a length of 5.
...
- *.df: PAGE_ID, WORD, FREQ, with exception of the first line corresponding to the article, which lists its name. This is a listing of a word document frequency where a document is considered to be a revision and the corpus is all revisions of an article.
- *.sentid_sent: PAGE_ID, SENTID, SENTENCE
- *.revid_sentid: PAGE_ID, REVID, UNSIMPLE_FLAG, SENTENCE_COUNT, SENTENCE_STREAM, COMMENT
- *.directed_sentid_sentid_w: PAGE_ID, REVISION1_ID, REVISION2_ID, SENTENCE1_ID, SENTENCE2_ID, ALIGNMENT_SCORE, PHRASE1_LENGTH, PHRASE1, PHRASE2_LENGTH2, PHRASE2
- *.index: This is an index for looing looking up particular page info. There is one of these for each of the types listed above, but it takes the same format: PAGE_ID, END_LINE_NUMBER, NUMBER_OF_LINES.
Two fields need additional explanation:
SENTENCE_STREAM: This is a comma seperated separated string of SENTID in the order they appear in the revision, with the character "P" used to indicate paragraph breaks.
UNSIMPLE_FLAG: This a flag for whether or not the revision was tagged with the "UNSIMPLE" group. Editors had the option of marking a file with this tag to indicate that it needed further simplification.
...