2

BERTで英検を解く

2 min read

BERT solves Eiken problems

Eiken (実用英語技能検定) is an English proficiency test conducted by a Japanese public-interest incorporated foundation (Link to wikipedia). One type of the questions in the test is a multiple choice problem to fill a blank in a sentence. For example:

My sister usually plays tennis (   ) Saturdays.

  1. by  2. on  3. with  4. at

Bob (   ) five friends to his party.

  1. made  2. visited  3. invited  4. spoke

In this notebook we solve this type of questions using pre-trained BERT models.

First, we use the masked language model, which is designed to guess a word in a blank in a sentence. A drawback of this approach is that the model cannot guess a word not included in its vocabulary set.

To handle unknown words, the second approach calculates perplexity scores of the sentences filled by choices. Since a lower perplexity score indicates the sentense is more "natural," we can pick the sentence with the lowest score as the answer.




Collecting openpyxl
  Downloading openpyxl-3.0.7-py2.py3-none-any.whl (243 kB)
     |████████████████████████████████| 243 kB 815 kB/s 
[?25hCollecting et-xmlfile
  Downloading et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)
Installing collected packages: et-xmlfile, openpyxl
Successfully installed et-xmlfile-1.1.0 openpyxl-3.0.7
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv


/opt/conda/lib/python3.7/site-packages/torchaudio/backend/utils.py:54: UserWarning: "sox" backend is being deprecated. The default backend will be changed to "sox_io" backend in 0.8.0 and "sox" backend will be removed in 0.9.0. Please migrate to "sox_io" backend. Please refer to https://github.com/pytorch/audio/issues/903 for the detail.
  '"sox" backend is being deprecated. '



Downloading:   0%|          | 0.00/480 [00:00<?, ?B/s]



Downloading:   0%|          | 0.00/331M [00:00<?, ?B/s]



Downloading:   0%|          | 0.00/899k [00:00<?, ?B/s]



Downloading:   0%|          | 0.00/456k [00:00<?, ?B/s]



Downloading:   0%|          | 0.00/1.36M [00:00<?, ?B/s]





[{'sequence': 'HuggingFace is creating a tool that the community uses to solve NLP tasks.',
  'score': 0.17927570641040802,
  'token': 3944,
  'token_str': ' tool'},
 {'sequence': 'HuggingFace is creating a framework that the community uses to solve NLP tasks.',
  'score': 0.11349428445100784,
  'token': 7208,
  'token_str': ' framework'},
 {'sequence': 'HuggingFace is creating a library that the community uses to solve NLP tasks.',
  'score': 0.05243517830967903,
  'token': 5560,
  'token_str': ' library'},
 {'sequence': 'HuggingFace is creating a database that the community uses to solve NLP tasks.',
  'score': 0.034935519099235535,
  'token': 8503,
  'token_str': ' database'},
 {'sequence': 'HuggingFace is creating a prototype that the community uses to solve NLP tasks.',
  'score': 0.028602516278624535,
  'token': 17715,
  'token_str': ' prototype'}]


Help on method __call__ in module transformers.pipelines.fill_mask:

__call__(*args, targets=None, top_k: Union[int, NoneType] = None, **kwargs) method of transformers.pipelines.fill_mask.FillMaskPipeline instance
    Fill the masked token in the text(s) given as inputs.
    
    Args:
        args (:obj:`str` or :obj:`List[str]`):
            One or several texts (or one list of prompts) with masked tokens.
        targets (:obj:`str` or :obj:`List[str]`, `optional`):
            When passed, the model will return the scores for the passed token or tokens rather than the top k
            predictions in the entire vocabulary. If the provided targets are not in the model vocab, they will be
            tokenized and the first resulting token will be used (with a warning).
        top_k (:obj:`int`, `optional`):
            When passed, overrides the number of predictions to return.
    
    Return:
        A list or a list of list of :obj:`dict`: Each result comes as list of dictionaries with the following keys:
    
        - **sequence** (:obj:`str`) -- The corresponding input with the mask token prediction.
        - **score** (:obj:`float`) -- The corresponding probability.
        - **token** (:obj:`int`) -- The predicted token id (to replace the masked one).
        - **token** (:obj:`str`) -- The predicted token (to replace the masked one).


RobertaForMaskedLM(
  (roberta): RobertaModel(
    (embeddings): RobertaEmbeddings(
      (word_embeddings): Embedding(50265, 768, padding_idx=1)
      (position_embeddings): Embedding(514, 768, padding_idx=1)
      (token_type_embeddings): Embedding(1, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): RobertaEncoder(
      (layer): ModuleList(
        (0): RobertaLayer(
          (attention): RobertaAttention(
            (self): RobertaSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): RobertaSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): RobertaIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): RobertaOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (1): RobertaLayer(
          (attention): RobertaAttention(
            (self): RobertaSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): RobertaSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): RobertaIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): RobertaOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (2): RobertaLayer(
          (attention): RobertaAttention(
            (self): RobertaSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): RobertaSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): RobertaIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): RobertaOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (3): RobertaLayer(
          (attention): RobertaAttention(
            (self): RobertaSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): RobertaSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): RobertaIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): RobertaOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (4): RobertaLayer(
          (attention): RobertaAttention(
            (self): RobertaSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): RobertaSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): RobertaIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): RobertaOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (5): RobertaLayer(
          (attention): RobertaAttention(
            (self): RobertaSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): RobertaSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): RobertaIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): RobertaOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
  )
  (lm_head): RobertaLMHead(
    (dense): Linear(in_features=768, out_features=768, bias=True)
    (layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    (decoder): Linear(in_features=768, out_features=50265, bias=True)
  )
)


[Problem(text='A: What is your {}?  B: Kazumi Suzuki.', choices=['hour', 'club', 'date', 'name'], answer='name'),
 Problem(text='I know Judy. She can {} French very well.', choices=['see', 'drink', 'speak', 'open'], answer='speak'),
 Problem(text="A: Are your baseball shoes in your room, Mike?  B: No, Mom. They're in my {} at school.", choices=['window', 'shop', 'locker', 'door'], answer='locker'),
 Problem(text='Mysister usually plays tennis {} Saturdays.', choices=['by', 'on', 'with', 'at'], answer='on'),
 Problem(text='My mother likes {}. She has many pretty ones in the garden.', choices=['sports', 'movies', 'schools', 'flowers'], answer='flowers'),
 Problem(text="Let's begin today's class. Open your textbooks to {} 22.", choices=['chalk', 'ground', 'page', 'minute'], answer='page'),
 Problem(text='Today is Wednesday. Tomorrow is {}.', choices=['Monday', 'Tuesday', 'Thursday', 'Friday'], answer='Thursday'),
 Problem(text='I usually read magazines {} home.', choices=['of', 'on', 'with', 'at'], answer='at'),
 Problem(text="A: It's ten o'clock, Jimmy. {} to bed.  B: All right, Mom.", choices=['Go', 'Sleep', 'Do', 'Sit'], answer='Go'),
 Problem(text="A: Do you live {} Tokyo?  B: Yes. It's a big city.", choices=['after', 'with', 'on', 'in'], answer='in')]


problem scores answer_position correct grade method
0 A: What is your {}? B: Kazumi Suzuki. name(0.245),nickname(0.101),surname(0.058),ans... 1 True 5 fill-mask-no-target
1 I know Judy. She can {} French very well. speak(0.858),read(0.053),learn(0.012),teach(0.... 1 True 5 fill-mask-no-target
2 A: Are your baseball shoes in your room, Mike?... room(0.408),locker(0.132),classroom(0.107),clo... 2 True 5 fill-mask-no-target
3 Mysister usually plays tennis {} Saturdays. on(0.953),every(0.021),through(0.007),until(0.... 1 True 5 fill-mask-no-target
4 My mother likes {}. She has many pretty ones i... roses(0.326),flowers(0.147),strawberries(0.070... 2 True 5 fill-mask-no-target
5 Let's begin today's class. Open your textbooks... page(0.285),chapter(0.171),Chapter(0.113),Page... 1 True 5 fill-mask-no-target
6 Today is Wednesday. Tomorrow is {}. Friday(0.293),Thursday(0.236),Sunday(0.168),Tu... 2 True 5 fill-mask-no-target
7 I usually read magazines {} home. at(0.945),from(0.026),back(0.023),around(0.001... 1 True 5 fill-mask-no-target
8 A: It's ten o'clock, Jimmy. {} to bed. B: All... Go(0.868),Get(0.039),Back(0.039),Went(0.009),G... 1 True 5 fill-mask-no-target
9 A: Do you live {} Tokyo? B: Yes. It's a big c... in(0.874),near(0.075),outside(0.043),around(0.... 1 True 5 fill-mask-no-target



problem scores answer_position correct grade method
0 A: What is your {}? B: Kazumi Suzuki. name(0.245),date(0.002),club(0.000),hour(0.000) 1 True 5 fill-mask-with-targets
1 I know Judy. She can {} French very well. speak(0.858),drink(0.000),see(0.000),open(0.000) 1 True 5 fill-mask-with-targets
2 A: Are your baseball shoes in your room, Mike?... locker(0.132),window(0.000),door(0.000),shop(0... 1 True 5 fill-mask-with-targets
3 Mysister usually plays tennis {} Saturdays. on(0.953),at(0.001),with(0.000),by(0.000) 1 True 5 fill-mask-with-targets
4 My mother likes {}. She has many pretty ones i... flowers(0.147),movies(0.000),sports(0.000),sch... 1 True 5 fill-mask-with-targets
5 Let's begin today's class. Open your textbooks... page(0.285),minute(0.000),chalk(0.000),ground(... 1 True 5 fill-mask-with-targets
6 Today is Wednesday. Tomorrow is {}. Friday(0.293),Thursday(0.236),Tuesday(0.105),M... 2 False 5 fill-mask-with-targets
7 I usually read magazines {} home. at(0.945),on(0.000),with(0.000),of(0.000) 1 True 5 fill-mask-with-targets
8 A: It's ten o'clock, Jimmy. {} to bed. B: All... Go(0.868),Sleep(0.000),Sit(0.000),Do(0.000) 1 True 5 fill-mask-with-targets
9 A: Do you live {} Tokyo? B: Yes. It's a big c... in(0.874),on(0.001),with(0.000),after(0.000) 1 True 5 fill-mask-with-targets





problem scores answer_position correct
0 My father is a {} of a sports club. He plays t... member(0.660),founder(0.068),patron(0.067),dir... 1 True
1 Mr. Clark told us many intesting {} about his ... stories(0.447),anecdotes(0.249),tales(0.187),t... 1 True
2 It's snowing a lot today, so please drive {}. accordingly(0.111),responsibly(0.091),carefull... -1 False
3 In spring, Jane likes to walk in her grandmoth... garden(0.659),yard(0.233),gardens(0.042),backy... 1 True
4 Many girls in my class have {} hair. curly(0.335),blonde(0.172),blond(0.074),facial... -1 False
5 A: Do you live in a city? B: No. I live in a ... town(0.515),city(0.139),village(0.099),suburb(... 1 True
6 I {} Nancy's notebook. It was on Mary's desk remember(0.180),remembered(0.115),borrowed(0.0... 4 True
7 Dennis went to Japan for a year in August. He ... said(0.822),waved(0.081),bid(0.039),says(0.011... 1 True
8 Jeff left the party at 8:00. He wanted to {} h... go(0.398),get(0.369),head(0.099),come(0.050),s... 2 True
9 Mom's lemon cake is not as good {} her chocola... as(0.998),for(0.001),than(0.000),with(0.000),a... 1 True
The specified target token ` coldly` does not exist in the model vocabulary. Replacing with `Ġcold`.
The specified target token ` busily` does not exist in the model vocabulary. Replacing with `Ġbus`.
problem scores answer_position correct
0 My father is a {} of a sports club. He plays t... member(0.660),group(0.000),festival(0.000),pic... 1 True
1 Mr. Clark told us many intesting {} about his ... stories(0.447),books(0.000),pictures(0.000),ma... 1 True
2 It's snowing a lot today, so please drive {}. slowly(0.036),cold(0.001),freely(0.000),bus(0.... 1 True
3 In spring, Jane likes to walk in her grandmoth... garden(0.659),wall(0.000),stone(0.000),sky(0.000) 1 True
4 Many girls in my class have {} hair. short(0.023),late(0.000),busy(0.000),slow(0.000) 1 True
5 A: Do you live in a city? B: No. I live in a ... town(0.515),hobby(0.000),holiday(0.000),ticket... 1 True
6 I {} Nancy's notebook. It was on Mary's desk found(0.077),stayed(0.000),stopped(0.000),went... 1 True
7 Dennis went to Japan for a year in August. He ... said(0.822),told(0.001),ended(0.000),hoped(0.000) 1 True
8 Jeff left the party at 8:00. He wanted to {} h... get(0.369),send(0.000),meet(0.000),put(0.000) 1 True
9 Mom's lemon cake is not as good {} her chocola... as(0.998),to(0.000),by(0.000),of(0.000) 1 True



problem scores answer_position correct
0 A: How do you make that potato dish? B: First... peel(0.218),roast(0.106),bake(0.099),boil(0.08... 4 True
1 Last summer, Hiroshi's family traveled around ... skiing(0.214),surfing(0.121),camping(0.098),fi... -1 False
2 Bob {} five friends to his party invites(0.555),invited(0.367),welcomes(0.031),... 2 True
3 A: John, you should go to bed soon. If you sta... graduate(0.145),crash(0.076),cry(0.051),die(0.... -1 False
4 A: Did you buy your father something special f... one(0.087),stove(0.038),spoon(0.033),dish(0.03... -1 False
5 I bought a new T-shirt for my brother, but I b... big(0.277),small(0.083),expensive(0.051),bulky... 5 True
6 Sarah saw some flowers by the road while she w... picked(0.319),bought(0.086),collected(0.085),g... 1 True
7 Jenny saw her grandparents {} the first time i... for(0.815),at(0.052),in(0.027),again(0.011),li... 1 True
8 A: I told my mother that I would be home by 7:... break(0.769),fulfill(0.066),violate(0.044),kee... 1 True
9 A: Don'y say anything to Dad about the surpris... anything(0.642),out(0.304),word(0.020),somethi... 2 True
The specified target token ` oversleep` does not exist in the model vocabulary. Replacing with `Ġovers`.
The specified target token ` apron` does not exist in the model vocabulary. Replacing with `Ġa`.
problem scores answer_position correct
0 A: How do you make that potato dish? B: First... boil(0.087),eat(0.003),hurt(0.000),care(0.000) 1 True
1 Last summer, Hiroshi's family traveled around ... abroad(0.019),inside(0.000),similar(0.000),oth... 1 True
2 Bob {} five friends to his party invited(0.367),made(0.000),visited(0.000),spok... 1 True
3 A: John, you should go to bed soon. If you sta... graduate(0.145),return(0.001),promise(0.000),o... -1 False
4 A: Did you buy your father something special f... ring(0.002),field(0.000),a(0.000),contact(0.000) -1 False
5 I bought a new T-shirt for my brother, but I b... tight(0.046),heavy(0.042),bright(0.002),clear(... 1 True
6 Sarah saw some flowers by the road while she w... picked(0.319),spent(0.000),guessed(0.000),wish... 1 True
7 Jenny saw her grandparents {} the first time i... for(0.815),from(0.003),over(0.001),out(0.000) 1 True
8 A: I told my mother that I would be home by 7:... break(0.769),sell(0.000),pass(0.000),lend(0.000) 1 True
9 A: Don'y say anything to Dad about the surpris... out(0.304),through(0.000),near(0.000),within(0... 1 True



problem scores answer_position correct
0 Jamie visited several {} buildings when he wen... ancient(0.177),Roman(0.109),historic(0.066),By... 1 True
1 Sally's French teacher told her to read an art... translate(0.944),paste(0.033),translated(0.007... 1 True
2 Henry likes living in the city because there a... woods(0.460),mountains(0.118),countryside(0.05... 3 True
3 A: Is it true that the things in this store on... extra(0.120),yen(0.100),postage(0.098),taxes(0... -1 False
4 When the bust was an hour late, one man shoute... angrily(0.495),insults(0.126),loudly(0.057),cu... 1 True
5 Firefighters have to {} people from buildings ... rescue(0.601),evacuate(0.232),remove(0.049),pu... 1 True
6 John loves the singer Ann May, and he cannot w... released(0.776),out(0.174),published(0.015),av... 1 True
7 The news that Ms. Kelly, the art teacher, was ... blew(0.197),swept(0.149),spread(0.141),came(0.... 3 True
8 A: I'm really nervous about acting in the play... stage(0.415),scene(0.078),screen(0.048),show(0... 1 True
9 Before Diane attended Professor Miller's {} at... lectures(0.595),lecture(0.120),classes(0.041),... 2 True
problem scores answer_position correct
0 Jamie visited several {} buildings when he wen... ancient(0.177),exact(0.000),responsible(0.000)... 1 True
1 Sally's French teacher told her to read an art... translate(0.944),throw(0.000),guide(0.000),con... 1 True
2 Henry likes living in the city because there a... countryside(0.057),image(0.000),experiment(0.0... 1 True
3 A: Is it true that the things in this store on... tax(0.011),data(0.000),total(0.000),waste(0.000) 1 True
4 When the bust was an hour late, one man shoute... angrily(0.495),tightly(0.000),partly(0.000),se... 1 True
5 Firefighters have to {} people from buildings ... rescue(0.601),weigh(0.000),produce(0.000),stam... 1 True
6 John loves the singer Ann May, and he cannot w... released(0.776),invented(0.000),trapped(0.000)... 1 True
7 The news that Ms. Kelly, the art teacher, was ... spread(0.141),stretched(0.000),served(0.000),s... 1 True
8 A: I'm really nervous about acting in the play... stage(0.415),screen(0.048),field(0.011),court(... 1 True
9 Before Diane attended Professor Miller's {} at... lecture(0.120),comment(0.000),furniture(0.000)... 1 True



problem scores answer_position correct
0 At first, the marketing department and the sal... jointly(0.257),entirely(0.139),separately(0.13... 2 True
1 Experts at the art gallery discovered that one... Pablo(0.455),genuine(0.041),fake(0.037),signed... 2 True
2 The musician Jimmy Baker had a lot of {} when ... money(0.189),fame(0.113),success(0.088),proble... -1 False
3 Mother Teresa helped many sick people and gave... orphans(0.279),humanity(0.114),children(0.082)... 2 True
4 As Liam waled down the dark street, he began t... feeling(0.477),impression(0.221),sense(0.122),... 5 True
5 Risa buys water that comes from a mountain str... nutrients(0.880),minerals(0.047),vitamins(0.02... 2 True
6 The lifeguard ran into the ocean to help a you... drowning(0.797),drowned(0.061),swimming(0.027)... 1 True
7 Yesterday was a hot day at the zoo, so Heather... putting(0.326),spilling(0.107),pouring(0.064),... 2 True
8 In the past, sailors had to use the stars to {... communicate(0.272),navigate(0.219),signal(0.07... 2 True
9 Daisuke's grandmother eats a lot of vegetables... improve(0.144),check(0.116),boost(0.089),asses... -1 False
The specified target token ` needlessly` does not exist in the model vocabulary. Replacing with `Ġneed`.
problem scores answer_position correct
0 At first, the marketing department and the sal... entirely(0.139),scientifically(0.000),violentl... 1 True
1 Experts at the art gallery discovered that one... genuine(0.041),severe(0.000),portable(0.000),l... 1 True
2 The musician Jimmy Baker had a lot of {} when ... hardship(0.002),concentration(0.000),membershi... 1 True
3 Mother Teresa helped many sick people and gave... humanity(0.114),generation(0.000),gravity(0.00... 1 True
4 As Liam waled down the dark street, he began t... sensation(0.018),property(0.000),feature(0.000... 1 True
5 Risa buys water that comes from a mountain str... minerals(0.047),operations(0.000),illustration... 1 True
6 The lifeguard ran into the ocean to help a you... drowning(0.797),proposing(0.000),converting(0.... 1 True
7 Yesterday was a hot day at the zoo, so Heather... spilling(0.107),maintaining(0.000),convincing(... 1 True
8 In the past, sailors had to use the stars to {... navigate(0.219),satisfy(0.000),respect(0.000),... 1 True
9 Daisuke's grandmother eats a lot of vegetables... preserve(0.019),replace(0.000),betray(0.000),i... 1 True



problem scores answer_position correct
0 A: Thanks for showing me the outline of your s... confusing(0.174),repetitive(0.051),boring(0.04... -1 False
1 Lisa went to the interview even though she tho... probability(0.391),likelihood(0.337),chance(0.... 1 True
2 It is sadly {} that, in developing counties, m... ironic(0.355),sad(0.098),true(0.081),unfortuna... 1 True
3 The explosion at the chemical factory {} great... inflicted(0.798),caused(0.093),wrought(0.036),... 1 True
4 Some say the best way to overcome a {} is to e... fear(0.577),stigma(0.036),dilemma(0.032),depre... -1 False
5 English classes at the university were require... excluded(0.375),exempted(0.224),barred(0.143),... 2 True
6 E-mail and text messaging have {} the way peop... changed(0.602),altered(0.146),transformed(0.10... 3 True
7 Some analysts think the new treaty on CO2 emis... milestone(0.416),breakthrough(0.238),landmark(... 1 True
8 Lying on the sunny beach with her husband on t... so(0.175),incredibly(0.166),very(0.063),extrem... -1 False
9 Nadine spends an hour thoroughly cleaning her ... clean(0.318),tidy(0.183),cleaner(0.109),cleane... -1 False
The specified target token ` barricade` does not exist in the model vocabulary. Replacing with `Ġbarric`.
The specified target token ` phobia` does not exist in the model vocabulary. Replacing with `Ġph`.
The specified target token ` officiated` does not exist in the model vocabulary. Replacing with `Ġoffic`.
The specified target token ` synthesized` does not exist in the model vocabulary. Replacing with `Ġsynthes`.
The specified target token ` disarmed` does not exist in the model vocabulary. Replacing with `Ġdis`.
The specified target token ` vigor` does not exist in the model vocabulary. Replacing with `Ġvig`.
The specified target token ` spotless` does not exist in the model vocabulary. Replacing with `Ġspot`.
problem scores answer_position correct
0 A: Thanks for showing me the outline of your s... redundant(0.006),subjective(0.000),distinct(0.... 1 True
1 Lisa went to the interview even though she tho... probability(0.391),credibility(0.000),contenti... 1 True
2 It is sadly {} that, in developing counties, m... ironic(0.355),superficial(0.000),indefinite(0.... 1 True
3 The explosion at the chemical factory {} great... inflicted(0.798),enhanced(0.000),perceived(0.0... 1 True
4 Some say the best way to overcome a {} is to e... temptation(0.002),famine(0.000),ph(0.000),barr... -1 False
5 English classes at the university were require... exempted(0.224),qualified(0.000),prosecuted(0.... 1 True
6 E-mail and text messaging have {} the way peop... transformed(0.105),dis(0.000),synthes(0.000),o... 1 True
7 Some analysts think the new treaty on CO2 emis... milestone(0.416),backlog(0.000),confession(0.0... 1 True
8 Lying on the sunny beach with her husband on t... profoundly(0.004),barely(0.000),harshly(0.000)... 1 True
9 Nadine spends an hour thoroughly cleaning her ... spot(0.000),rugged(0.000),impartial(0.000),min... -1 False



problem scores answer_position correct
0 Cell phones have become a permanent {} in mode... fixture(0.932),feature(0.011),necessity(0.011)... 1 True
1 Colin did not have enough money to pay for the... increments(0.440),excess(0.375),installments(0... 3 True
2 When she asked her boss for a raise, Melanie's... calm(0.040),soft(0.034),low(0.031),quiet(0.028... -1 False
3 The religious sect established a {} in a rural... sanctuary(0.225),monastery(0.219),temple(0.073... -1 False
4 The famous reporter was fired for {} another j... criticizing(0.301),copying(0.131),publishing(0... -1 False
5 Now that the local steel factory has closed do... bustling(0.132),abandoned(0.120),struggling(0.... -1 False
6 The ambassador's failure to attend the ceremon... embarrassment(0.558),insult(0.365),injustice(0... -1 False
7 US border guards managed to {} the escaped pri... detain(0.213),apprehend(0.205),arrest(0.067),c... 2 True
8 Anthony enjoyed his first day at his new job. ... welcoming(0.102),upbeat(0.077),positive(0.076)... -1 False
9 A: I just learned I've been {} to second violi... promoted(0.438),relegated(0.294),upgraded(0.08... 2 True
The specified target token ` dispositions` does not exist in the model vocabulary. Replacing with `Ġdispos`.
The specified target token ` enactments` does not exist in the model vocabulary. Replacing with `Ġenact`.
The specified target token ` speculations` does not exist in the model vocabulary. Replacing with `Ġspec`.
The specified target token ` garish` does not exist in the model vocabulary. Replacing with `Ġgar`.
The specified target token ` jovial` does not exist in the model vocabulary. Replacing with `Ġj`.
The specified target token ` pompous` does not exist in the model vocabulary. Replacing with `Ġpomp`.
The specified target token ` diffident` does not exist in the model vocabulary. Replacing with `Ġdiff`.
The specified target token ` dirge` does not exist in the model vocabulary. Replacing with `Ġdir`.
The specified target token ` prelude` does not exist in the model vocabulary. Replacing with `Ġpre`.
The specified target token ` commune` does not exist in the model vocabulary. Replacing with `Ġcommun`.
The specified target token ` alleviating` does not exist in the model vocabulary. Replacing with `Ġallev`.
The specified target token ` plagiarizing` does not exist in the model vocabulary. Replacing with `Ġplagiar`.
The specified target token ` inoculating` does not exist in the model vocabulary. Replacing with `Ġinoc`.
The specified target token ` beleaguering` does not exist in the model vocabulary. Replacing with `Ġbe`.
The specified target token ` elucidation` does not exist in the model vocabulary. Replacing with `Ġeluc`.
The specified target token ` affront` does not exist in the model vocabulary. Replacing with `Ġaff`.
The specified target token ` impasse` does not exist in the model vocabulary. Replacing with `Ġimp`.
The specified target token ` ultimatum` does not exist in the model vocabulary. Replacing with `Ġult`.
The specified target token ` pillage` does not exist in the model vocabulary. Replacing with `Ġpill`.
The specified target token ` exalt` does not exist in the model vocabulary. Replacing with `Ġex`.
The specified target token ` acclimate` does not exist in the model vocabulary. Replacing with `Ġacc`.
The specified target token ` congenial` does not exist in the model vocabulary. Replacing with `Ġcongen`.
The specified target token ` delirious` does not exist in the model vocabulary. Replacing with `Ġdel`.
The specified target token ` measly` does not exist in the model vocabulary. Replacing with `Ġmeas`.
The specified target token ` implausible` does not exist in the model vocabulary. Replacing with `Ġimpl`.
The specified target token ` jeopardized` does not exist in the model vocabulary. Replacing with `Ġjeopard`.
The specified target token ` stowed` does not exist in the model vocabulary. Replacing with `Ġst`.
problem scores answer_position correct
0 Cell phones have become a permanent {} in mode... fixture(0.932),rupture(0.000),stint(0.000),cla... 1 True
1 Colin did not have enough money to pay for the... installments(0.163),spec(0.000),dispos(0.000),... 1 True
2 When she asked her boss for a raise, Melanie's... j(0.000),diff(0.000),gar(0.000),pomp(0.000) -1 False
3 The religious sect established a {} in a rural... repository(0.000),commun(0.000),pre(0.000),dir... -1 False
4 The famous reporter was fired for {} another j... plagiar(0.000),be(0.000),allev(0.000),inoc(0.000) -1 False
5 Now that the local steel factory has closed do... defunct(0.001),aspiring(0.000),volatile(0.000)... 1 True
6 The ambassador's failure to attend the ceremon... imp(0.000),aff(0.000),ult(0.000),eluc(0.000) -1 False
7 US border guards managed to {} the escaped pri... apprehend(0.205),pill(0.000),acc(0.000),ex(0.000) 1 True
8 Anthony enjoyed his first day at his new job. ... congen(0.000),del(0.000),impl(0.000),meas(0.000) -1 False
9 A: I just learned I've been {} to second violi... relegated(0.294),reiterated(0.000),st(0.000),j... 1 True





Downloading:   0%|          | 0.00/764 [00:00<?, ?B/s]



Downloading:   0%|          | 0.00/3.25G [00:00<?, ?B/s]



Downloading:   0%|          | 0.00/1.04M [00:00<?, ?B/s]



Downloading:   0%|          | 0.00/456k [00:00<?, ?B/s]



Downloading:   0%|          | 0.00/1.36M [00:00<?, ?B/s]




problem scores answer_position correct
0 A: What is your {}? B: Kazumi Suzuki. name(4.421),date(5.132),club(5.409),hour(5.515) 1 True
1 I know Judy. She can {} French very well. speak(3.975),see(4.961),drink(5.339),open(5.466) 1 True
2 A: Are your baseball shoes in your room, Mike?... locker(3.825),shop(4.090),door(4.124),window(4... 1 True
3 Mysister usually plays tennis {} Saturdays. on(6.950),at(7.842),by(8.166),with(8.233) 1 True
4 My mother likes {}. She has many pretty ones i... flowers(3.275),movies(4.204),sports(4.250),sch... 1 True
5 Let's begin today's class. Open your textbooks... page(3.568),minute(4.388),chalk(4.695),ground(... 1 True
6 Today is Wednesday. Tomorrow is {}. Thursday(2.731),Friday(2.820),Tuesday(3.075),M... 1 True
7 I usually read magazines {} home. at(4.812),on(6.230),of(6.704),with(6.817) 1 True
8 A: It's ten o'clock, Jimmy. {} to bed. B: All... Go(3.637),Sleep(4.108),Do(4.159),Sit(4.297) 1 True
9 A: Do you live {} Tokyo? B: Yes. It's a big c... in(3.254),on(3.652),with(3.861),after(3.912) 1 True
problem scores answer_position correct
0 My father is a {} of a sports club. He plays t... member(3.035),group(4.094),festival(4.196),pic... 1 True
1 Mr. Clark told us many intesting {} about his ... stories(4.840),pictures(5.429),magazines(5.593... 1 True
2 It's snowing a lot today, so please drive {}. slowly(3.082),busily(3.700),coldly(3.714),free... 1 True
3 In spring, Jane likes to walk in her grandmoth... garden(3.513),stone(4.185),wall(4.230),sky(4.275) 1 True
4 Many girls in my class have {} hair. short(3.469),late(4.467),slow(4.653),busy(5.382) 1 True
5 A: Do you live in a city? B: No. I live in a ... town(2.939),hobby(3.440),holiday(3.561),ticket... 1 True
6 I {} Nancy's notebook. It was on Mary's desk found(4.157),stopped(4.780),went(4.977),stayed... 1 True
7 Dennis went to Japan for a year in August. He ... said(3.226),told(3.726),ended(3.983),hoped(3.999) 1 True
8 Jeff left the party at 8:00. He wanted to {} h... get(2.797),meet(3.353),send(3.444),put(3.580) 1 True
9 Mom's lemon cake is not as good {} her chocola... as(3.458),of(4.667),by(4.727),to(4.805) 1 True
problem scores answer_position correct
0 A: How do you make that potato dish? B: First... boil(3.264),eat(3.363),hurt(3.521),care(3.718) 1 True
1 Last summer, Hiroshi's family traveled around ... abroad(4.146),inside(4.269),other(4.620),simil... 1 True
2 Bob {} five friends to his party invited(5.150),made(6.476),spoke(6.545),visite... 1 True
3 A: John, you should go to bed soon. If you sta... oversleep(2.627),return(2.889),graduate(3.012)... 1 True
4 A: Did you buy your father something special f... apron(3.104),ring(3.225),field(3.478),contact(... 1 True
5 I bought a new T-shirt for my brother, but I b... tight(3.286),heavy(3.405),bright(3.512),clear(... 1 True
6 Sarah saw some flowers by the road while she w... picked(2.792),guessed(3.328),spent(3.331),wish... 1 True
7 Jenny saw her grandparents {} the first time i... for(2.761),over(3.653),out(3.665),from(3.704) 1 True
8 A: I told my mother that I would be home by 7:... break(2.799),pass(3.114),sell(3.139),lend(3.226) 1 True
9 A: Don'y say anything to Dad about the surpris... out(3.642),through(4.512),near(4.575),within(4... 1 True
problem scores answer_position correct
0 Jamie visited several {} buildings when he wen... ancient(3.093),exact(3.583),responsible(3.771)... 1 True
1 Sally's French teacher told her to read an art... translate(3.196),guide(3.870),throw(4.024),con... 1 True
2 Henry likes living in the city because there a... countryside(2.423),experiment(2.904),image(2.9... 1 True
3 A: Is it true that the things in this store on... tax(3.296),data(3.545),total(3.574),waste(3.582) 1 True
4 When the bust was an hour late, one man shoute... angrily(3.245),secretly(3.659),tightly(3.678),... 1 True
5 Firefighters have to {} people from buildings ... rescue(2.607),produce(3.223),weigh(3.473),stam... 1 True
6 John loves the singer Ann May, and he cannot w... released(3.660),invented(4.074),divided(4.186)... 1 True
7 The news that Ms. Kelly, the art teacher, was ... spread(3.004),stretched(3.238),served(3.492),s... 1 True
8 A: I'm really nervous about acting in the play... stage(3.899),screen(4.051),field(4.055),court(... 1 True
9 Before Diane attended Professor Miller's {} at... lecture(3.845),comment(4.133),furniture(4.239)... 1 True
problem scores answer_position correct
0 At first, the marketing department and the sal... entirely(2.498),needlessly(2.740),violently(2.... 1 True
1 Experts at the art gallery discovered that one... genuine(3.079),portable(3.520),severe(3.677),l... 1 True
2 The musician Jimmy Baker had a lot of {} when ... hardship(3.326),concentration(3.565),membershi... 1 True
3 Mother Teresa helped many sick people and gave... humanity(2.718),generation(3.180),gravity(3.18... 1 True
4 As Liam waled down the dark street, he began t... sensation(3.421),feature(4.111),translation(4.... 1 True
5 Risa buys water that comes from a mountain str... minerals(3.101),operations(3.727),illustration... 1 True
6 The lifeguard ran into the ocean to help a you... drowning(3.241),proposing(4.074),converting(4.... 1 True
7 Yesterday was a hot day at the zoo, so Heather... spilling(3.160),maintaining(3.649),arguing(3.8... 1 True
8 In the past, sailors had to use the stars to {... navigate(3.120),respect(3.493),satisfy(3.554),... 1 True
9 Daisuke's grandmother eats a lot of vegetables... preserve(3.041),replace(3.377),betray(3.537),i... 1 True
problem scores answer_position correct
0 A: Thanks for showing me the outline of your s... redundant(3.329),subjective(3.454),distinct(3.... 1 True
1 Lisa went to the interview even though she tho... probability(2.479),credibility(3.207),contenti... 1 True
2 It is sadly {} that, in developing counties, m... ironic(3.586),superficial(4.055),indefinite(4.... 1 True
3 The explosion at the chemical factory {} great... inflicted(3.238),perceived(3.704),enhanced(3.8... 1 True
4 Some say the best way to overcome a {} is to e... phobia(2.733),temptation(3.107),barricade(3.29... 1 True
5 English classes at the university were require... exempted(3.016),qualified(3.583),prosecuted(3.... 1 True
6 E-mail and text messaging have {} the way peop... transformed(4.060),synthesized(4.406),disarmed... 1 True
7 Some analysts think the new treaty on CO2 emis... milestone(2.537),vigor(2.962),backlog(3.035),c... 1 True
8 Lying on the sunny beach with her husband on t... profoundly(3.576),barely(3.864),harshly(4.001)... 1 True
9 Nadine spends an hour thoroughly cleaning her ... spotless(3.993),rugged(4.641),impartial(4.788)... 1 True
problem scores answer_position correct
0 Cell phones have become a permanent {} in mode... fixture(3.340),rupture(4.229),clasp(4.242),sti... 1 True
1 Colin did not have enough money to pay for the... installments(3.012),dispositions(3.537),specul... 1 True
2 When she asked her boss for a raise, Melanie's... jovial(3.002),diffident(3.208),pompous(3.282),... 2 False
3 The religious sect established a {} in a rural... commune(3.466),repository(3.890),prelude(4.160... 1 True
4 The famous reporter was fired for {} another j... plagiarizing(2.819),beleaguering(3.335),inocul... 1 True
5 Now that the local steel factory has closed do... defunct(3.112),volatile(3.456),aspiring(3.502)... 1 True
6 The ambassador's failure to attend the ceremon... affront(3.527),ultimatum(3.723),impasse(3.901)... 1 True
7 US border guards managed to {} the escaped pri... apprehend(3.144),acclimate(3.557),pillage(3.58... 1 True
8 Anthony enjoyed his first day at his new job. ... congenial(2.469),delirious(2.613),measly(2.855... 1 True
9 A: I just learned I've been {} to second violi... relegated(3.020),jeopardized(3.433),reiterated... 1 True





grade problem_number problem scores(fill-mask-no-target) correct(fill-mask-no-target) scores(fill-mask-with-targets) correct(fill-mask-with-targets) scores(perplexity) correct(perplexity)
0 5 1 A: What is your {}? B: Kazumi Suzuki. name(0.245),nickname(0.101),surname(0.058),ans... True name(0.245),date(0.002),club(0.000),hour(0.000) True name(4.421),date(5.132),club(5.409),hour(5.515) True
1 5 2 I know Judy. She can {} French very well. speak(0.858),read(0.053),learn(0.012),teach(0.... True speak(0.858),drink(0.000),see(0.000),open(0.000) True speak(3.975),see(4.961),drink(5.339),open(5.466) True
2 5 3 A: Are your baseball shoes in your room, Mike?... room(0.408),locker(0.132),classroom(0.107),clo... True locker(0.132),window(0.000),door(0.000),shop(0... True locker(3.825),shop(4.090),door(4.124),window(4... True
3 5 4 Mysister usually plays tennis {} Saturdays. on(0.953),every(0.021),through(0.007),until(0.... True on(0.953),at(0.001),with(0.000),by(0.000) True on(6.950),at(7.842),by(8.166),with(8.233) True
4 5 5 My mother likes {}. She has many pretty ones i... roses(0.326),flowers(0.147),strawberries(0.070... True flowers(0.147),movies(0.000),sports(0.000),sch... True flowers(3.275),movies(4.204),sports(4.250),sch... True
method fill-mask-no-target fill-mask-with-targets perplexity
grade
1 4 5 9
pre1 6 8 10
2 8 10 10
pre2 9 10 10
3 7 8 10
4 8 10 10
5 10 9 10



Discussion

コメントにはログインが必要です。