Have you published a disruptive paper? New machine-learning tool helps you check

Scientists in the US have unveiled a new machine-learning tool that, they claim, can identify disruptive scientific breakthroughs. They say their method, which assesses how much a paper reshapes its field, is better than other techniques at spotting such disruptions even if they are simultaneously discovered by independent research groups (Sci. Adv. 12 eadx3420).

The work examined 55 million papers listed by Web of Science and the American Physical Society (APS) published between 1893 and 2019. The papers were mapped using a machine-learning technique known as neural embedding, with each publication represented by two vector points. The first vector characterizes the body of work the paper builds on while the second represents the research it inspires.

Papers that disrupt tend to cause future research to depart significantly from previous work in the field, making these “past” and “future” vectors diverge sharply. The greater the divergence, the higher the paper’s so-called Embedding Disruptiveness Measure (EDM) score.

The team, based at Indiana and Binghamton universities, tested their EDM technique against Nobel-prize-winning papers and milestone publications as selected by APS editors. The EDM identified these landmark contributions as being highly disruptive.

The researchers discovered that the EDM was more consistent at spotting such papers than similar metrics, such as the “disruption index”, which focuses more on a publication’s closest citations. While this makes it sensitive to individual citations, it can miss the bigger picture, the researchers found.

The team discovered that the 10 papers with the biggest difference between the EDM and the disruption index were all examples of “simultaneous disruption”. This is where multiple papers have independently reached the same conclusion, or scientists have published their work across publications. Citations that linked these simultaneous disruptive papers weakened their disruption index.

One notable example is the two 1974 papers announcing the discovery of the J/ψ meson. As both groups cited each other, the disruption index ranked these publications in the bottom 1% of disruptive papers while the EDM placed them both in the top 10%. A similar pattern was seen for the two 1964 papers – one by Peter Higgs and the other by François Englert and Robert Brout – on the Higgs mechanism.

The team claims that the EDM also provides a new way to detect simultaneous discoveries, finding that papers that report the same breakthrough tend to be cited in similar contexts by later work, meaning their “future” vectors cluster together.

“By having more accurate metrics, we can actually investigate where the disruption is happening in the map of science,” says data scientist Sadamori Kojaku from Binghamton University.

The researchers say their tool could help science funding and policy to drive transformative breakthroughs. “It can have significant implications for science policy and it’s also helpful for prioritizing funding,” adds Kojaku. “We now have the quantitative metrics to investigate at which stage of research the disruptive work occurs and matters most.”

The post Have you published a disruptive paper? New machine-learning tool helps you check appeared first on Physics World.